Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning

# Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
Source: [https://arxiv.org/html/2607.07761](https://arxiv.org/html/2607.07761)
1ORCID of Qi Peng: https://orcid\.org/0000\-0001\-7747\-3293∗ORCID of Changmeng Zheng: https://orcid\.org/0000\-0002\-2945\-8248

\\equalcont

These authors contributed equally to this work\.

\\equalcont

These authors contributed equally to this work\.

\\equalcont

These authors contributed equally to this work\.

\[1\]Changmeng Zheng

1\]\\orgnameThe Hong Kong Polytechnic University,\\orgaddress\\cityHong Kong,\\countryChina

2\]\\orgnameHong Kong University,\\orgaddress\\cityHong Kong,\\countryChina

3\]\\orgnameSouth China University of Technology,\\orgaddress\\cityGuangzhou,\\countryChina

4\]\\orgnameUniversity of Toronto,\\orgaddress\\cityToronto,\\countryCanada

5\]\\orgnamePeking Union Medical College Hospital,\\orgaddress\\cityBeijing,\\countryChina

6\]\\orgnameWest China Hospital, Sichuan University,\\orgaddress\\citySichuan,\\countryChina

###### Abstract

Large language models \(LLMs\) have emerged as important tools in healthcare, showing growing potential for clinical reasoning and patient care\. This survey examines recent progress in medical LLMs, focusing on reasoning applications and requirements\. We present a dual\-view approach that connects clinical practice with computational methods\. On the clinical side, we establish a five\-level competency scheme following Miller’s Pyramid, progressing from knowledge recall to dynamic case management\. On the computational side, we link deductive, inductive, and abductive reasoning patterns to common medical goals and tasks\. We also introduce a benchmark dataset spanning five levels of medical reasoning capability and report results on 18 state\-of\-the\-art models, revealing that medical specialist models excel in diagnosis\-centric tasks while general models lead in decision support and dialogue\. We conclude by discussing current progress and open challenges, including data limitations, hallucination, and grounding issues, and outline directions toward safer, more reliable, and workflow\-ready systems\.

###### keywords:

Large Language Model, Medical LLMs, Dual\-View Approach, Miller’s Pyramid, Reasoning

![Refer to caption](https://arxiv.org/html/2607.07761v1/x1.png)Figure 1:A general pipeline for LLM reasoning on medicine\.## 1Introduction

The medical field faces growing pressure to deliver timely and accurate decisions while managing increasingly large volumes of data and constantly evolving evidence\. Clinicians must skillfully combine patient history, physical examination findings, test results, and imaging data, while carefully weighing risks and preferences and acting under significant uncertainty\. Although traditional toolsaljabri2022towards;neves2014survey;kumar2013content;lin2025cohortprovide valuable support for data retrieval and rule execution, they often struggle when information is incomplete or conflicting, or when decisions depend critically on sequence, context, and complex trade\-offs\. Meanwhile, parallel advances in data availability and computational power have driven rapid progress in large language modelsA\_survey\_of\_large\_language\_models;qiu2023pre\. These LLMs can effectively read and generate clinical text, connect to external sources, and engage in conversational interactions that naturally fit clinical workflowsliang2025answer\. Consequently, this has led to widespread interest in applying LLMs across the entire care pathway, spanning from pre\-visit triage and symptom assessment through diagnosissun2024medical;shao2025mcanet;nie2025reparameterized, treatment planningbaig2024accuracy;sehar2026automatic, documentationseo2024evaluation, and follow\-up carechow2016inter\.

This survey specifically focuses on medical reasoning capabilities rather than general language processing applications\. Our primary goal is toclarify what kinds of reasoning are actually needed in clinical practice, how current model designs map to these specific needs, and how meaningful progress should be measured\. To achieve this, we adopt two complementary perspectives\.From the clinical view, we systematically extend Miller’s PyramidThe\_assessment\_of\_clinical\_skills/competence/performanceinto five distinct levels: knowledge recognition and normalization; information classification and triage; causal reasoning and comprehensive diagnosis; decision support and individualized recommendations; and dynamic interaction in complex clinical scenarios\. This framework provides a structured way to align specific tasks with expected competency levels along the care continuum\.From the computational view, we organize existing methods by reasoning type and systematically link deductive, inductive, and abductive reasoning patterns to concrete medical objectives such as symptom normalizationMedNorm, risk predictionyang2024ensuring, differential diagnosishirosawa2023diagnostic, and treatment selectionbaig2024accuracy\. Importantly, mixed reasoning settings that combine these patterns are common in realistic clinical workflows and are therefore treated explicitly in our analysis\.

Building on these perspectives, we map datasets and benchmarks according to both reasoning type and clinical competency level, covering resources for entity grounding, clinical triage, diagnostic reasoning, question answering, imaging\-informed tasks, documentation, and dialogue management\. Furthermore, we thoroughly review model designs and training strategies currently used in practice, including instruction tuning with expert datazhang2023pmc, chain\-of\-thought and long\-form reasoningchen2025towards, search\- and tree\-guided reasoningzhang2025llama, retrieval\-augmented generation with knowledge bases and literaturebora2024systematic, tool use and agent workflowsliu2025autoct;zheng2024picture, and multimodal integration for text and imagesboehm2022harnessing;feng2023towards\. We also discuss evaluation methodologies that extend beyond simple accuracy, encompassing factuality, reasoning completeness, internal consistency, evidence use, and uncertainty handling, utilizing both expert review and automated metrics\.

To support standardized measurement across the field, we construct a comprehensive benchmark dataset with five levels of medical reasoning capability, containing 5,000 carefully curated samples \(1,000 per level\)\. Through systematic evaluation of 18 state\-of\-the\-art models on this benchmark, we observe several important trends\. Notably, medical specialist models tend to perform better on diagnosis\-centric tasks, while high\-capacity general models often excel in decision support, multi\-turn dialogue, and summarization tasks\. Interestingly, length\-of\-stay prediction and other structured temporal outcomes remain challenging for all systems tested\. Our analysis reveals that model size alone does not fully explain performance differences; rather, instruction quality, domain\-specific data, and reasoning training appear equally important\. Based on these findings, a practical deployment approach would involve routing diagnosis\-heavy queries to specialized medical models while leveraging general models for dialogue and supportive tasks\.

This paper’s contributions can be summarized as follows:

- •We propose a dual\-view framework that systematically links clinical competence levels with core reasoning types and aligns them with common medical tasks\.
- •We introduce a standardized five\-level benchmark with 5,000 samples and report detailed results on 18 state\-of\-the\-art models, yielding insights on where medical and general models excel and where they struggle\.
- •We provide a thorough review of training and inference methods for medical reasoning, including instruction tuning, chain\-of\-thought, retrieval and tool use, agent workflows, and multimodal integration, while outlining evaluation criteria that assess both answers and reasoning steps\.
- •We systematically discuss current limitations and risks, including data gaps, hallucination, grounding challenges, and calibration issues, and suggest concrete directions for safer and more workflow\-ready systems\.

Organization: The remainder of this paper is structured as follows\. Section II provides background and presents our proposed taxonomy linking clinical competencies with reasoning types\. Section III systematically surveys datasets and benchmarks organized by reasoning type and clinical level\. Section IV reviews model designs and reasoning paradigms, including chain\-of\-thought, long\-form, search\-guided, retrieval\-augmented, multimodal, and agentic methods\. Section V introduces our five\-level benchmark and reports detailed results for 18 models with analysis of observed trends\. Section VI discusses current challenges and limitations\. Section VII outlines recent advances and future directions\. Finally, Section VIII concludes with key insights and implications\.

## 2Background and Taxonomy

The foundation of LLM reasoning in medicine rests upon three interconnected pillars: the evolution of large language models and their core technologies, the theoretical understanding of reasoning processes, and the established frameworks for assessing clinical competence\. This section provides comprehensive background knowledge essential for understanding the current state and future potential of medical reasoning systems\.

### 2\.1Overview of Medical Large Language Models

The landscape of artificial intelligence has been redefined by the scaling of general\-purpose large language models \(LLMs\) like GPT\-3brown2020language, PaLMchowdhery2023palm, and GPT\-4achiam2023gpt, which demonstrated emergent capabilities in reasoning and generalization\. This progression catalyzed a parallel evolution in the medical domain, transitioning from early BERT\-based adaptations like ClinicalBERThuang2019clinicalbertand BioBERTlee2020biobertto expert\-level generative systems\. Recent milestones, such as MedPaLMsinghal2023largeand Med\-Geminisaab2024capabilities, have successfully harnessed model scaling to achieve near\-human performance on standardized medical examinations and complex clinical reasoning tasks\. To effectively adapt these powerful architectures for clinical utility, researchers employ specialized training and prompting strategies\. Domain adaptation is primarily achieved through either training models from scratch on biomedical corpora, as exemplified by PubMedBERTgu2021domainand BioGPTluo2022biogpt, or through continuous pre\-training of general checkpoints, as seen in ClinicalGPTwang2023clinicalgpt\. Furthermore, instruction tuningwei2021finetunedhas emerged as a critical alignment mechanism, where models are fine\-tuned on expert\-curated medical datasets to ensure safe and professional responses\. Beyond training, inference\-time strategies like Chain\-of\-Thought \(CoT\) promptingwei2022chainhave proven indispensable\. By guiding models to decompose complex clinical scenarios into step\-by\-step reasoning paths, CoT aligns algorithmic processing with the structured diagnostic logic inherent to medical practice\.

### 2\.2Classical Reasoning Paradigms

The paradigm of dividing reasoning into deductive reasoning, inductive reasoning, and abductive reasoning was first proposed by the renowned American philosopher Charles Sanders Peirce\. This framework ingeniously utilizes the relationships among “cases”, “rules”, and “results” to clearly elucidate the essential differences between different types of reasoning\. This classification framework has been employed in teaching reasoning\-related fields for many years and has been widely adopted by the academic community\.

Deductive Reasoningderives specific results from established general rules and known cases \(Rule \+ Case→\\rightarrowResult\)\. It is characterized byirrefutability: if the premises are true, the conclusion must necessarily be trueDeductive\_reasoning\_2\. For instance, given the rule “High fever indicates infection” and the case “This patient has a fever,” the result “This patient is infected” is logically certain\. While deduction ensures reliability in fields like mathematics, it does not generate new information beyond what is already implicit in the premises\.

Deductive ReasoningpremisesCaseThis patient has a fever of 39°C\.RuleHigh fever indicates infection\.conclusionResultThis patient is infected\.Table 1:Deductive Reasoning, Barbara SyllogismInductive Reasoninggeneralizes new rules from observed cases and results \(Case \+ Result→\\rightarrowRule\)\. Unlike deduction, its conclusions aredefeasibleand probabilistic rather than certainAbductive\_and\_inductive\_reasoning:\_background\_and\_issues\. For example, observing that “Many patients with high fever improved after antibiotics” allows us to induce the general rule “High fever indicates bacterial infection\.” Although this process can generate new knowledge necessary for scientific discovery, the conclusions remain subject to revision if counter\-examples arise\.

Inductive ReasoningpremisesCaseMany patients have a fever of 39°C\.ResultThey improved after antibiotics\.conclusionRuleHigh fever indicates infection\.Table 2:Inductive Reasoning, Barbara SyllogismAbductive Reasoninginfers the most plausible case \(cause\) from known rules and observed results \(Rule \+ Result→\\rightarrowCase\)\. It shares the defeasible nature of induction but focuses on explanation rather than generalization\. In a clinical context, this corresponds to diagnostic reasoning: given the result “The patient is infected” and the rule “High fever indicates infection,” a physician abductively infers “The patient likely has a fever” as the explanation\. This form of reasoning is fundamental to medical diagnosis, enabling physicians to hypothesize underlying causes from symptomsAbductive\_and\_inductive\_reasoning:\_background\_and\_issues\.

Abductive ReasoningpremisesRuleHigh fever indicates infection\.ResultThis patient is infected\.conclusionCaseThis patient has a fever of 39°C\.Table 3:Abductive Reasoning, Barbara SyllogismAfter introducing the general concept of reasoning and its classical classification paradigms \(deductive reasoning, inductive reasoning, and abductive reasoning\), we can see that these reasoning theories not only constitute the core framework of human cognition and artificial intelligence, but are also widely applied and extended in specific domains\. In the medical field, reasoning processes often involve complex clinical decision\-making, diagnostic inference, and treatment planning, which essentially depend on the ability to derive conclusions from symptoms, test results, and medical knowledge\. Applying general reasoning paradigms to medical contexts not only helps understand physicians’ thought processes, but also provides a solid theoretical foundation for developing LLM medical reasoning taxonomy\. Next, we will introduce the classical clinical competency grading framework to further understand the unique characteristics of medical reasoning\.

### 2\.3Levels of Clinical Competence

To better understand the reasoning capabilities of Large Language Models in the medical domain, we establish a hierarchical classification of LLM reasoning from the perspective of medical competency grading\. This classification not only helps define the capability boundaries of LLMs in medical reasoning, but also enables the assessment and regulation of their safety and effectiveness in actual clinical applications\. Below, we provide a brief introduction to Miller’s PyramidThe\_assessment\_of\_clinical\_skills/competence/performance, the mainstream framework for medical competency grading that serves as background knowledge\.

Miller’s Pyramid, introduced by George Miller in 1990, provides a foundational framework for understanding and assessing clinical competence\. The pyramid consists of four levels: “Knows”, “Knows How”, “Shows How”, and “Does”\. This framework has profound implications for understanding how LLMs might be integrated into medical practice and how their capabilities should be evaluated\.

Knowsas the base of the pyramid, representing factual knowledge and information recall\. At this level, assessment focuses on whether individuals can remember and recognize relevant medical facts, terminology, and concepts\.

Knows Howas the second Level of the pyramid, representing the ability to apply and interpret knowledge\. At this level, the assessment focuses on whether individuals can understand and apply medical knowledge to solve problems, analyze situations, or formulate plans\.

Shows Howas the third Level of the pyramid, representing skill demonstration in simulated environments\. At this level, the assessment focus is on whether individuals can demonstrate clinical skills and procedural operations under controlled conditions \(such as laboratory or simulated scenarios\)\.

Doesas the top of the pyramid, representing authentic performance in real\-world environments\. At this level, the evaluation focuses on whether individuals can independently execute tasks, make decisions, and provide patient care in actual clinical practice\.

Modern medical education and assessment have extended Miller’s framework to address contemporary challenges in healthcare delivery and medical practice\. These extensions are particularly relevant to understanding how LLMs might be integrated into clinical practice and what additional competencies they might need to develop\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x2.png)Figure 2:Overview of LLMs and datasets under different types of medical reasoning\. The diagram organizes representative datasets and models according to four reasoning paradigms: deductive, inductive, abductive, and mixed\. In addition, different types of LLMs are categorized, and datasets are further grouped by corresponding medical tasks\.
### 2\.4Taxonomy of Medical Reasoning Capabilities

Building upon the theoretical foundations of reasoning and clinical competence frameworks, we propose a comprehensive taxonomy for LLM medical reasoning capabilities\. This taxonomy integrates classical reasoning paradigms with practical medical applications through two complementary organizational structures\.

#### 2\.4\.1Reasoning Type Classification

Fig\.[2](https://arxiv.org/html/2607.07761#S2.F2)illustrates the mapping of medical tasks to four fundamental reasoning types\.Deductive reasoning tasksinvolve applying established medical knowledge and guidelines to specific situations, such as symptom normalization and test result interpretation\.Inductive reasoning tasksidentify patterns in clinical data to make predictions, including length of stay prediction and readmission risk assessment\.Abductive reasoning tasksgenerate and evaluate explanatory hypotheses for observed clinical phenomena, central to diagnostic processes\.Mixed reasoning taskscombine multiple reasoning paradigms to address complex clinical scenarios requiring integrated decision\-making\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x3.png)Figure 3:The proposed five\-level competency hierarchy for evaluating LLM capabilities in medical reasoning\. The hierarchy extends Miller’s Pyramid to the context of large language models, with ascending levels from basic medical knowledge recognition and normalization \(Level 1\) to dynamic interaction and complex scenario management \(Level 5\)\. Higher levels build upon competencies from the lower tiers, reflecting increasing reasoning sophistication and integration in clinical contexts\.
#### 2\.4\.2Five\-Level Competency Hierarchy

Fig\.[3](https://arxiv.org/html/2607.07761#S2.F3)presents our proposed five\-level competency hierarchy that extends Miller’s Pyramid to the context of large language models\. This framework provides a structured approach to understanding and evaluating LLM capabilities in medical reasoning contexts, with each level corresponding to specific stages in Miller’s classical framework for assessing clinical competence:

Level 1: Medical Knowledge Recognition and Normalization \(Knows\)forms the foundation, corresponding to Miller’s “Knows” level\. This involves accurate recognition, interpretation, and standardization of medical concepts and terminology\. Systems must demonstrate ability to identify medical entities, understand semantic relationships, and map natural language to structured concepts in standardized medical knowledge bases\. Typical tasks include symptom normalization, medical entity recognition, and medical terminology standardization\.

Level 2: Information Classification and Clinical Triage \(Knows How\)aligns with Miller’s “Knows How” level, involving categorization and prioritization of clinical information to support decision\-making and resource allocation\. This requires understanding of disease severity, urgency indicators, and institutional capabilities\. Systems at this level can perform department triage prediction, urgency detection, and preliminary diagnostic classification, demonstrating knowledge of how medical principles should be applied\.

Level 3: Causal Reasoning and Comprehensive Diagnosis \(Shows How\)corresponds to Miller’s “Shows How” level, encompassing sophisticated diagnostic reasoning that integrates multiple information sources \(including medical history, physical signs, laboratory results, and imaging\) to reach comprehensive clinical conclusions\. Systems must integrate diverse clinical information, perform causal reasoning, consider multiple diagnostic hypotheses, and handle uncertainty appropriately\. This includes comprehensive diagnostic reasoning, test result interpretation, longitudinal diagnosis, and disease progression analysis\.

Level 4: Clinical Decision Support and Personalized Recommendations \(Shows How/Does\)bridges Miller’s “Shows How” and “Does” levels, focusing on treatment planning and clinical decision support\. This requires integration of evidence\-based medicine with individual patient factors and preferences to provide personalized recommendations\. Systems at this level can perform medication recommendation, surgical/treatment planning, treatment outcome prediction, follow\-up suggestions, and readmission risk assessment, demonstrating near\-clinical performance capabilities\.

Level 5: Dynamic Interaction and Complex Scenario Management \(Does\)represents the highest level of competence, corresponding to Miller’s “Does” level\. This encompasses dynamic, interactive reasoning in complex clinical scenarios that require real\-time adaptation and multi\-faceted decision\-making\. Systems must continuously collect information, adjust reasoning pathways, and manage complex cases through multi\-turn diagnostic conversations, dynamic disease management, discharge summary generation, and comprehensive clinical question answering with decision support\.

Each ascending level builds upon competencies from lower tiers, reflecting increasing reasoning sophistication and clinical integration\. This hierarchical structure not only provides theoretical grounding through its alignment with Miller’s Pyramid but also offers practical utility for researchers and practitioners in systematically evaluating and developing medical reasoning systems\. The framework emphasizes the progression from foundational knowledge \(“Knows”\) through application understanding \(“Knows How”\), demonstrated competence \(“Shows How”\), to actual clinical performance \(“Does”\), mirroring the developmental trajectory expected in medical education and practice\.

## 3Datasets and Benchmarks

### 3\.1General Workflow for Constructing Medical Datasets

The construction of high\-quality medical datasets is a prerequisite for advancing reasoning capabilities of large language models \(LLMs\) in healthcare\. Unlike general\-domain corpora, medical datasets must meet strict requirements regarding data authenticity, privacy protection, and domain specificity\. A general workflow can be summarized in five stages: data acquisition, preprocessing and normalization, annotation, validation, and benchmarking preparation, which is shown in Fig\.[4](https://arxiv.org/html/2607.07761#S3.F4)\.

#### 3\.1\.1Data Acquisition

Medical data can originate from diverse sources, including electronic health records \(EHRs\), clinical notes, diagnostic reports, biomedical literature, medical imaging, and patient\-generated health data\. To acquire such data typically requires collaboration with healthcare institutions and strict adherence to ethical and regulatory frameworks\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x4.png)Figure 4:The general workflow for constructing medical datasets\.
#### 3\.1\.2Preprocessing and Normalization

Raw medical data is often heterogeneous and noisy, containing free\-text descriptions, structured tables, imaging formats, and laboratory values\. Preprocessing involves de\-identification of patient information, removal of artifacts, and normalization into standardized formats\. This step ensures interoperability and facilitates downstream reasoning tasks\.

#### 3\.1\.3Annotation

Accurate and consistent annotation is crucial for supervised learning and evaluation\. Annotation typically requires domain experts such as physicians or trained medical annotators\. Depending on the task, annotation may involve entity labeling \(e\.g\., symptoms, medications, diagnoses\), relation extraction \(e\.g\., drug–drug interactions\), or reasoning\-related tasks such as assigning diagnostic hypotheses or treatment recommendations\. In this step, inter\-annotator agreement measurement is essential for ensuring reliability\.

#### 3\.1\.4Validation and Quality Control

Rigorous validation is necessary to guarantee dataset quality and usability\. This involves internal consistency checks, cross\-validation against external references \(e\.g\., clinical guidelines\), and expert review\. Statistical analyses are often conducted to ensure representativeness across patient demographics, disease categories, and clinical settings\. In addition, bias detection and mitigation are critical to support generalizable reasoning, such as addressing demographic imbalance or hospital\-specific artifacts\.

#### 3\.1\.5Benchmarking Preparation

Finally, datasets must be partitioned and standardized for benchmarking\. Typical practices include dividing data into training, validation, and test sets; designing task\-specific evaluation metrics; and defining challenge tasks aligned with reasoning paradigms\. Public release requires careful documentation of dataset scope, limitations, and licensing terms\. Increasingly, benchmark design also incorporates multi\-institutional test sets and adversarial evaluation scenarios to assess robustness and fairness\.

In summary, medical dataset construction is not merely a data collection exercise but a structured pipeline balancing clinical fidelity, ethical considerations, and technical requirements\. Based on this workflow, reasoning\-specific and competency\-level benchmarks can be systematically developed\.

Table 4:Overview of datasets in different medical reasoning capabilities levels\.CapabilitiesTaskDatasetModalityTask TypeNumber of SamplesAnswer FormLanguageLevel1:KnowledgerecognitionandnormalizationSymptomNormalizationCADECCADECTxtNER, RE∼\\sim1,250 postsFree\-FormEnglishUMLSUMLSTxtNER, RE\-Free\-FormEnglishMedNormMedNormTxtNER,Concept Normalization27,979descriptionsFree\-FormEnglishMedical NamedEntity RecognitionCBLUEzhang\-etal\-2022\-cblueTxtMulti\-task NLU\-Free\-form / labelsChineseNCBI DiseaseNCBITxtDisease NER &Concept Normalization793 abstractsSpan \+Concept IDEnglishAbbreviationDisambiguationin Clinical TextsMeDALMeDALTxtAbbr\. Disambig\.14,393,619 articlesOpen\-text / labelEnglishCASICASITxtAbbr\. Disambig\.440 abbreviationsLabelEnglishClinical\-ieClinical\-ieTxtInformation ExtractionDepends onsubtasksSpan /Structured outputEnglishMedicationName NormalizationOpenFDAkass2016openfdaTxtRegulatory Retrieval\-Structured JSONEnglishLevel2:InformationclassificationandpreliminaryjudgmentDepartmentTriageMedDGliu2022meddgTxtDialogue,Response Generation17,864 dialogues,385,951 utterancesOpenChineseCMIDCMIDTxtIntent Understanding∼\\sim12K dialoguesMC\-QAChineseEmergencyService–Triage Application1TabClassification∼\\sim1267 instancesOpenEnglishPreliminaryDiagnosis PredictionDDXPlusfansi2022ddxplusTabSymptom Detection∼\\sim1\.3M patientsOpenEnglishMIMIC\-IV\-Ext DiReCTMIMIC\-IV\-Ext\-DiReCTTxtDiagnostic Reasoning511 clinical notesOpenEnglishDisease Diagnosis2TxtDiagnostic Reasoning∼\\sim2,000instancesOpenEnglishAcuity/UrgencyDetectionMC\-MEDMC\-MEDImg,TxtPrediction118,385visitsOpenEnglishMIETICMIETICTxtTriage Classification9,629 triage casesGen\.EnglishLevel3:CausalreasoningandcomprehensivediagnosisDiagnosticReasoningHeart Disease Dataheart\-disease\-45TxtPrediction303 instancesOpenEnglishCDSLCDSLTxtPrediction4,479 patientsOpenEnglishOCT3ImgClassification84,495 imagesClassificationEnglishChestX\-ray14ChestXray\-NIHCCImgDisease Detection112,120 imagesOpenEnglishTest ResultInterpretationDiabeticRetinopathy Detection4ImgClassification∼\\sim35,000imagesClassificationEnglishRSNAPneumonia Detection5ImgDetection∼\\sim30,000imagesOpenEnglishSLAKEliu2021slakeImg, TxtQA14,028 QA pairsOpenEN & CNVQA\-RADlau2018datasetImg, TxtQA315 images3\.5K QA pairsOpenEnglishPATHVQAhe2020pathvqaImg, TxtQA4,998 images32,799 QA pairsOpenEnglishLongitudinalDiagnostic ReasoningMIT\-BIHMIT\-BIH\-Arrhythmia\-DatabaseTxtDetection48 instancesClassificationEnglishLevel4:ClinicalDecisionSupportandIndividualizedRecommendationClinical QA& Decision SupportemrKBQAemrKBQATxtQA940K questionOpenEnglishMedDQAMedDQATxtQA2000 imagesMC\-QAChineseemrQAemrQATxtQA1,957,835 QA pairs,1,225,369 QL pairsOpenEnglishMedAgentsBenchtang2025medagentsbenchTxtQAseven establishedmedical datasetsFree\-FormEnglishReasonMedsun2025reasonmedTxtQA370k examplesFree\-FormEnglishMedicationRecommendationDrugBankDrugBankTxtInformation Extraction10,794 drugs, 1413413drug–drug interactionsFree\-FormEnglishPharmGKBPharmGKBTxtRelation Extraction\-Free\-FormEnglishRxNormRxNormTxtDrug Normalization\>\>100K conceptsFree\-FormEnglishTreatmentOutcome PredictionLung Cancer Datalung\_cancer\_62TxtSurvival Prediction32instancesOpenEnglishKaggle DiabetesPrediction Dataset6TxtPrediction768instancesOpenEnglishLength ofStay PredictionHospital Lengthof Stay Dataset7TxtPrediction∼\\sim100KinstancesOpenEnglishReadmissionRisk PredictionDiabetes 130\-USHospitals \(1999–2008\)diabetes\_130\-us\_hospitals\_for\_years\_1999\-2008\_296TxtClassification101,766patient recordsOpenEnglishFollow\-upRecommendationCLIPCLIPTxtMulti\-aspect extractivesummarization718 dischargeFree\-FormEnglishLevel5:DynamicInteractionandComplexSceneManagementDialogue\-basedReasoningMed\-QuADMedQuADTxtQA∼\\sim47K QA pairsOpenEnglishMedQAMedQATxtQA61,097 questionsMC\-QAEN & CNMedMCQAMedMCQATxtQA∼\\sim194K questionsMC\-QAEnglishPubMedQAPubMedQATxtQA211\.3k QA pairsOpenEnglishMultiMedQAsinghal2025towardTxtQA∼\\sim208k QA pairsMC\-QAEnglishHealthBencharora2025healthbenchTxtQA5,000 conversationsFree\-FormEnglishLiveQALiveMedQA2017TxtQA634 QA pairsOpenEnglishMedicationQAMedicationQATxtQA674 QA pairsOpenEnglishChinese MedicalDialogue Model8TxtDialogue, QA∼\\sim4K dialogsOpenChineseTCM Pre\-trained DatasetTCMChatTxtPretraining∼\\sim100M tokensGen\.Chinese3MDBenchsviridov20253mdbenchImg, TxtDialogue3013 casesFree\-FormEnglishDischargeSummary GenerationDischarge\-MeDischarge‑MeTxtSummarization109,168 visitsGen\.EnglishMedDecMedDecTxtMedical DecisionExtraction451 dischargesummariesFree\-FormEnglish2012 i2b2 TemporalRelations Dataset2012\-i2b2\-Temporal\-Relations\-DatasetTxtExtraction\-Free\-FormEnglishDiSCQDiSCQTxtQA2K\+ questionsFree\-formEnglish

- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8

### 3\.2Datasets Categorized by Reasoning Types

Medical reasoning tasks can be systematically organized according to the type of reasoning\. Here we review datasets by four reasoning types: deductive, inductive, abductive, and mixed reasoning\.

#### 3\.2\.1Deductive Reasoning Datasets

Deductive reasoning datasets emphasize the application of established medical knowledge, clinical guidelines, and standardized rules to specific patient cases\. Unlike inductive or abductive paradigms, deductive reasoning is deterministic in nature, aiming to derive logically valid conclusions when the premises are correct\. Accordingly, datasets in this category are often rule\-grounded, designed to align with established medical ontologies and evidence\-based protocols\.

A defining characteristic of deductive reasoning datasets is their structured and standardized nature\. The data typically includes explicit mappings between clinical inputs, such as symptom descriptions, laboratory results, or medication names\. For example, patient\-reported symptoms in free\-text may need to be normalized to standardized vocabularies such as SNOMED CT or ICD codes, or laboratory values may be interpreted relative to reference ranges and diagnostic thresholds\. These tasks demand precision and consistency, as small errors in normalization or interpretation can propagate downstream and compromise clinical decision\-making\.

Representative deductive reasoning datasets exemplify structured, rule\-based alignment with clinical standards\. For instance, the Diabetic Retinopathy Detection dataset111[https://www\.kaggle\.com/c/diabetic\-retinopathy\-detection/data](https://www.kaggle.com/c/diabetic-retinopathy-detection/data)provides retinal fundus images labeled with diabetic retinopathy severity, requiring models to interpret imaging findings against established diagnostic criteria\. Similarly, the RSNA Pneumonia Detection dataset222[https://www\.kaggle\.com/c/rsna\-pneumonia\-detection\-challenge/data](https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/data)offers annotated chest X\-rays for pneumonia presence, demanding models to identify abnormalities and map them to clinical diagnoses\.

#### 3\.2\.2Inductive Reasoning Datasets

Inductive reasoning datasets focus on identifying patterns in large\-scale clinical data and generalizing these patterns to make probabilistic predictions about individual patients or populations\. Unlike deductive tasks that rely on fixed guidelines, inductive datasets are inherently data\-driven, capturing statistical regularities in patient outcomes, treatment responses, and disease trajectories\.

Such datasets are usually derived from electronic health records \(EHRs\), intensive care unit \(ICU\) monitoring systems, or longitudinal patient databases\. They often contain a mix of structured variables alongside unstructured clinical notes\. The richness of these datasets enables predictive modeling, but also introduces challenges such as heterogeneity, imbalance, and institution\-specific artifacts\.

Representative inductive reasoning datasets capture statistical patterns in heterogeneous clinical data\. For example, the Emergency Service – Triage Application dataset333[https://www\.kaggle\.com/datasets/ilkeryildiz/emergency\-service\-triage\-application](https://www.kaggle.com/datasets/ilkeryildiz/emergency-service-triage-application)includes patient complaints, nursing assessments, triage levels, and department assignments, requiring models to predict triage severity\.

#### 3\.2\.3Abductive Reasoning Datasets

Abductive reasoning datasets are designed to support the generation and evaluation of explanatory hypotheses for observed clinical phenomena\. In contrast to deductive datasets and inductive datasets, abductive datasets aim to capture the diagnostic reasoning process, where multiple plausible explanations for patient presentations must be considered and weighed\.

These datasets often involve complex, multi\-source clinical information such as patient history, symptoms, imaging findings, and laboratory test results\. They require models to reason under uncertainty and to prioritize hypotheses based on plausibility and supporting evidence\. Unlike deductive tasks with definitive mappings, abductive datasets frequently admit multiple valid answers, reflecting the ambiguity inherent in real\-world diagnosis\.

Representative abductive reasoning datasets reflect diagnostic tasks that require generating and weighing multiple plausible explanations\. The Heart Disease dataset \(UCI\)heart\-disease\-45provides structured patient features such as blood pressure, ECG, and blood sugar, with outputs indicating disease presence, supporting inference of likely diagnoses from heterogeneous clinical signals\. ChestX\-ray14ChestXray\-NIHCCcontains over 100,000 chest X\-rays annotated for 14 thoracic conditions, requiring models to interpret imaging findings and propose probable disease explanations under uncertainty\.

#### 3\.2\.4Mixed Reasoning Datasets

Mixed reasoning datasets combine elements of deductive, inductive, and abductive reasoning, reflecting the hybrid and multi\-step nature of real\-world clinical practice\. A physician may begin by normalizing symptoms into standardized concepts \(deductive\), identify risk patterns from prior cases \(inductive\), and then generate plausible diagnostic hypotheses under uncertainty \(abductive\)\. Datasets in this category are therefore particularly valuable for evaluating whether LLMs can integrate multiple reasoning strategies in a coherent and clinically meaningful way\.

Such datasets are usually broad in scope, spanning diverse task types such as medical question answering, clinical dialogue, and exam\-style problem solving\. They often require systems to first extract structured information, then apply probabilistic prediction, and finally reason about diagnostic or therapeutic options\.

Representative mixed reasoning datasets highlight the integration of deductive, inductive, and abductive processes\. For example, CBLUEzhang\-etal\-2022\-cblue, a Chinese EMR entity recognition dataset, requires models to normalize clinical concepts using medical dictionaries \(deductive\) while generalizing entity patterns through statistical learning \(inductive\)\. NCBI DiseaseNCBIprovides PubMed abstracts annotated with disease entities mapped to standard ontologies, combining rule\-based normalization with contextual pattern recognition\.

### 3\.3Datasets Categorized by Levels of Medical Competency

To empirically ground the proposed five\-level competency hierarchy, this section reviews existing datasets as practical benchmarks for each stage of medical reasoning\. We systematically map available resources to the taxonomy, demonstrating how data characteristics evolve to match the increasing sophistication of the hierarchy: from foundational entity normalization \(Level 1\) and structured triage \(Level 2\), to complex causal diagnosis \(Level 3\) and personalized decision support \(Level 4\), culminating in dynamic, multi\-turn interaction scenarios \(Level 5\)\. This organization explicitly connects theoretical competencies with their corresponding evaluation protocols, highlighting the specific data requirements necessary to assess AI capabilities at each stratum of the adapted Miller’s Pyramid\.

#### 3\.3\.1Level 1: Medical Knowledge Recognition and Normalization

Datasets at this foundational level focus on the recognition, interpretation, and normalization of medical concepts from unstructured clinical or biomedical text\. They correspond to the most basic reasoning capabilities, enabling systems to accurately identify mentions of diseases, symptoms, medications, and procedures, and to map them to standardized terminologies\. This process ensures that heterogeneous clinical language can be reliably transformed into structured, interoperable knowledge\. Precision in lexical and semantic mapping is critical, as errors at this stage may cascade and compromise higher\-order reasoning tasks\.

A specific feature of datasets at this level is their strong alignment with biomedical ontologies and controlled vocabularies\. They are designed to test whether models can bridge free\-text variability with standardized representations, such as SNOMED CT, ICD, or UMLS concepts\. The emphasis lies not only on entity detection but also on disambiguation and normalization, ensuring consistency across different contexts of use\.

Representative datasets at this level emphasize concept recognition and normalization\. CADECCADEC, derived from medical forum posts, annotates drugs, adverse reactions, symptoms, and diseases, linking them to controlled vocabularies\. MedNormMedNormstandardizes symptoms and adverse drug events from social media, requiring models to detect informal mentions and align them with UMLS concepts\.

#### 3\.3\.2Level 2: Information Classification and Clinical Triage

Datasets at this level target the categorization, prioritization, and contextual interpretation of clinical information to support decision\-making and resource allocation\. Unlike datasets in level 1 that focus on concept recognition, Level 2 datasets assess whether systems can apply medical knowledge to evaluate the severity, urgency, or relevance of patient information\. This requires models to interpret clinical notes, laboratory results, or symptom reports in context and to assign appropriate classifications such as triage levels, department referrals, or risk strata\.

The feature of Level 2 datasets is their focus on structured decision labels derived from expert judgments or established protocols\. The data is often organized to facilitate evaluation of classification performance across multiple clinical dimensions, including urgency \(e\.g\., emergent vs\. non\-emergent\), specialty assignment \(e\.g\., cardiology vs\. gastroenterology\), and risk stratification \(e\.g\., high vs\. low readmission probability\)\. These datasets provide benchmarks for testing whether LLMs can move beyond entity recognition to context\-aware reasoning, integrating multiple pieces of information to reach actionable conclusions\.

Representative datasets for level 2 evaluate context\-aware classification and preliminary diagnosis\. DDXPlusfansi2022ddxplusmaps structured symptom descriptions to multiple possible diagnoses\. MIMIC\-IV\-Ext DiReCTMIMIC\-IV\-Ext\-DiReCTextends MIMIC\-IV with clinician\-annotated SOAP notes and reasoning trees, challenging models to reconstruct likely discharge diagnoses from incomplete patient records\. The Disease Diagnosis Dataset444[https://www\.kaggle\.com/datasets/s3programmer/disease\-diagnosis\-dataset](https://www.kaggle.com/datasets/s3programmer/disease-diagnosis-dataset)provides simulated patient cases with demographics, symptoms, vital signs, and annotated diagnoses with severity, testing models’ ability to classify both disease type and urgency\.

#### 3\.3\.3Level 3: Causal Reasoning and Comprehensive Diagnosis

Datasets at this level emphasize the integration of diverse clinical information to support comprehensive diagnostic reasoning\. Unlike Level 1 and Level 2 datasets, which focus on concept recognition or classification, Level 3 datasets require systems to consider multiple data sources, such as patient history, laboratory tests, imaging reports, and symptom trajectories, and generate well\-reasoned conclusions about potential diagnoses\. This involves causal inference, temporal reasoning, and evaluation of competing hypotheses under uncertainty, reflecting the complexity of real\-world clinical decision\-making\.

A defining characteristic of Level 3 datasets is their multi\-modal and multi\-step nature\. Cases are often structured as complex clinical scenarios or narratives, where correct interpretation depends on understanding relationships between findings, temporal progression of symptoms, and underlying pathophysiology\. Annotations frequently include primary diagnoses, differential diagnoses, and relevant supporting evidence, providing rich supervision for evaluating system reasoning and explanatory capabilities\.

Representative Level 3 datasets highlight multi\-modal integration and complex diagnostic reasoning\. OCT \(Optical Coherence Tomography\)555[https://www\.kaggle\.com/datasets/paultimothymooney/kermany2018](https://www.kaggle.com/datasets/paultimothymooney/kermany2018)provides large\-scale retinal images for detecting diseases such as macular degeneration, diabetic retinopathy, and glaucoma, requiring models to infer plausible diagnoses from imaging data under uncertainty\. CDSL \(COVID\-19 Data Shared Learning\)CDSLcombines vital signs, laboratory tests, and imaging data from COVID\-19 patients, demanding integration of heterogeneous inputs to determine infection status and severity\.

#### 3\.3\.4Level 4: Clinical Decision Support and Personalized Recommendations

Datasets at this level focus on treatment planning and clinical decision support, requiring systems to integrate evidence\-based guidelines with individual patient characteristics\. Unlike earlier levels, which emphasize recognition, classification, or diagnostic reasoning, Level 4 datasets test whether models can recommend personalized interventions, account for contraindications, and predict treatment outcomes\. These datasets are crucial for evaluating a system’s ability to move from diagnostic insight to actionable clinical guidance\.

In the datasets in level 4, they combine the patient\-specific features with structured clinical knowledge\. The data often includes demographics, comorbidities, laboratory and imaging results, and historical treatment responses, linked to recommended therapeutic options and expected outcomes\. This allows evaluation of whether systems can not only select guideline\-concordant treatments but also tailor recommendations to individual patient contexts, reflecting real\-world complexity in clinical decision\-making\.

Representative Level 4 datasets emphasize treatment planning and clinical decision support through mixed reasoning\. emrKBQAemrKBQAlinks physician questions, logical forms, and structured patient records, requiring integration of knowledge with patient\-specific data to generate accurate recommendations\. Similarly, EHRSQLlee2022ehrsqlpresents a text\-to\-SQL benchmark for question answering on electronic health records, requiring precise structured reasoning\. MedMCQAMedMCQApresents large\-scale multiple\-choice medical questions, testing both deductive knowledge recall and abductive contextual inference\.

#### 3\.3\.5Level 5: Dynamic Interaction and Complex Scenario Management

Datasets at this highest competency level evaluate dynamic, interactive reasoning in complex clinical scenarios that involve evolving patient conditions and multi\-faceted decision\-making\. Unlike Level 4 datasets, which focus on static treatment planning, Level 5 datasets test whether systems can adapt reasoning strategies in real time, coordinate multiple aspects of care, and manage multi\-system clinical interactions\. This level represents the most sophisticated form of medical reasoning, closely resembling the full spectrum of clinical practice\.

The defining feature of Level 5 datasets is their emphasis on multi\-turn tasks\. Cases often simulate real\-world clinical workflows, including diagnostic interviews, care coordination among multiple providers, and longitudinal monitoring of patient responses\. Annotated outcomes may include multi\-step decisions, patient\-centered care adjustments, and dynamic prioritization under changing conditions\.

Level 5 datasets test dynamic, interactive reasoning in clinical scenarios\. Med\-QuADMedQuADprovides multi\-turn medical dialogues for interpreting evolving information and inferring potential causes, while the Chinese Medical Dialogue Model666[https://www\.kaggle\.com/datasets/thedevastator/chinese\-medical\-dialogue\-model](https://www.kaggle.com/datasets/thedevastator/chinese-medical-dialogue-model)simulates patient–doctor conversations, requiring iterative information gathering and reasoning refinement\.

## 4Paradigms of Medical LLM Reasoning

Recent advances in Large Language Model \(LLM\) reasoning have shown remarkable potential in the medical domain, where tasks often require complex, multi\-step, and explainable reasoning\. In this section, we first present the evolutionary timeline of medical LLMs in Fig\.[5](https://arxiv.org/html/2607.07761#S4.F5), followed by an overview of the methodological categories of representative general\-purpose and medical LLMs in Fig\.[6](https://arxiv.org/html/2607.07761#S4.F6)\. We then review the major reasoning paradigms employed in medical LLMs, as illustrated in Fig\.[7](https://arxiv.org/html/2607.07761#S4.F7)and[8](https://arxiv.org/html/2607.07761#S4.F8), including Chain\-of\-Thought reasoning, Long\-CoT reasoning, search\-guided reasoning, retrieval\-augmented reasoning, and multimodal reasoning\. Fig\.[7](https://arxiv.org/html/2607.07761#S4.F7)provides a comprehensive mapping of representative medical LLMs against our proposed five\-level competency hierarchy and their underlying methodologies\. The heatmap reveals a clear distributional trend that mirrors the hierarchy’s increasing complexity: while the majority of surveyed models demonstrate robust proficiency in foundational tasks \(Levels 1 and 2\), represented by dense coverage in the upper rows, capability saturation drops significantly at the advanced stages \(Levels 4 and 5\)\. Crucially, the figure makes the connection between competency and methodology explicit; models achieving high\-level proficiency \(e\.g\., Level 5: Dynamic Interaction\) exhibit a strong correlation with advanced reasoning paradigms such as Long\-CoT and Agentic workflows, whereas models limited to lower levels predominantly rely on standard Chain\-of\-Thought or basic retrieval strategies\. This visual evidence underscores that ascending the competency hierarchy requires a commensurate evolution in algorithmic sophistication\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x5.png)Figure 5:An evolutionary timeline illustrating the progression of LLM\-based methods in medical AI\.![Refer to caption](https://arxiv.org/html/2607.07761v1/x6.png)Figure 6:Different methodological categories of medical LLMs\.![Refer to caption](https://arxiv.org/html/2607.07761v1/x7.png)Figure 7:The evolution history of medical LLM reasoning\.### 4\.1Chain\-of\-Thought Reasoning

Chain\-of\-Thought \(CoT\)wei2022chainreasoning can be formalized as a process where an LLM generates a sequence of linear reasoning steps,R=\(r1,r2,…,rn\)R=\(r\_\{1\},r\_\{2\},\\dots,r\_\{n\}\), which are interleaved with the final answer,AA\. The entire process is usually a single, continuous generation prompted by “Let’s think step by step”\.

The output, which includes both the reasoning steps and the final answer, can be represented as a single output stream produced by a modelℳ\\mathcal\{M\}on a given inputℐ\\mathcal\{I\}\. The formalization can be expressed as:

\(r1,r2,…,rn,A\)=ℳ​\(ℐ\)\.\(r\_\{1\},r\_\{2\},\\dots,r\_\{n\},A\)=\\mathcal\{M\}\(\\mathcal\{I\}\)\.\(1\)
In this framework, each reasoning steprir\_\{i\}and the final answerAAare generated sequentially, while each step is entirely dependent on the previous steps, which can be viewed as an auto\-regressive process:

r1\\displaystyle r\_\{1\}=ℳ​\(ℐ\)\\displaystyle=\\mathcal\{M\}\(\\mathcal\{I\}\)\(2\)r2\\displaystyle r\_\{2\}=ℳ​\(ℐ,r1\)\\displaystyle=\\mathcal\{M\}\(\\mathcal\{I\},r\_\{1\}\)⋮\\displaystyle\\vdotsA\\displaystyle A=ℳ​\(ℐ,r1,r2,…,rn\)\.\\displaystyle=\\mathcal\{M\}\(\\mathcal\{I\},r\_\{1\},r\_\{2\},\\dots,r\_\{n\}\)\.
In the medical domain, LLMs, such as GPT\-4achiam2023gpt, Med\-PaLMsinghal2023large, Med\-PaLM 2singhal2025toward, Med\-Geminisaab2024capabilities, MedGemmasellergren2025medgemmaand etc\., all demonstrate powerful CoT reasoning capabilities, generalizing well across various medical tasks\. However, beyond these LLMs with inherent CoT features, there are also several works trying to enhance their CoT reasoning ability\. For instance, MedCoTliu2024medcotproposes a hierarchical, multi\-expert CoT approach for Medical VQA, where an initial specialist produces a reasoning chain, a follow\-up specialist validates it\. MEDQA\-OPENnachane2024fewintroduces a modified open\-ended version of MedQA with clinician\-verified reasoning answers, and proposes a novel few\-shot CoT prompting scheme that mimics incremental clinical reasoning\. Besides, Med\-REFLyang2025medconstructs preference CoT pairs via the Tree\-of\-Thoughtyao2023treeprompting strategy and applies Direct Preference Optimizationrafailov2023directto train the reasoning capability of the LLM\.

### 4\.2Long Chain\-of\-Thought Reasoning

Building upon the Chain\-of\-Thought reasoning, Long Chain\-of\-Thought \(Long\-CoT\) reasoning takes a step further by encouraging or enabling the model to explore the problem space via significantly more extensive and detailed reasoning processesguo2025deepseek\. While standard CoT might involve a few concise linear exploration of the problem space, Long\-CoT is characterized by its depth, verbosity, and the inclusion of sub\-steps, explorations of alternative paths, self\-correction, and explicit verificationchen2025towards\.

The formalization of Long\-CoT retains a similar structure as standard CoT, but with an emphasis on the properties of the reasoning processesRl​o​n​g=\(r1,r2,…,rm\)R\_\{long\}=\(r\_\{1\},r\_\{2\},\\dots,r\_\{m\}\), wherem≫nm\\gg n\. Generally, Long\-CoT reasoning differs previous CoT in the following aspects:

- •Increased Length and Granularity: The number of reasoning steps is substantially larger\. Each steprir\_\{i\}might be more granular, breaking down a single logical leap from standard CoT into multiple, smaller, and more explicit deductions\.
- •Structural Complexity: The reasoning chainRl​o​n​gR\_\{long\}is not strictly linear\. It can include branching, self\-correction, and verification processes that help recursively explore the problem space\.
- •Enhanced Information Synthesis: The longer process allows the model to more thoroughly synthesize information from the inputℐ\\mathcal\{I\}connect disparate concepts, and manage a larger working memory of intermediate facts and sub\-problems\.

Owing to enhanced granularity in exploring the problem space, Long\-CoT reasoning LLMs, such as OpenAI o1jaech2024openai, DeepSeek\-R1guo2025deepseek, and QWQ\-32Bqwq32b, have demonstrated the effectiveness of Long\-CoT reasoning in solving complex commonsense and mathematical problems\. Notably, DeepSeek\-R1 firstly leverages a two stage training pipeline, combining Cold\-start Supervised Fine\-tuning \(SFT\) with Group Relative Policy Optimization \(GRPO\), a reinforcement learning \(RL\) framework to encourage deeper, multi\-step reasoning paths, thereby enhancing both accuracy and interpretability\.

In the medical domain, the Long\-CoT reasoning paradigm has catalyzed the development of specialized LLMs that prioritize interpretability and diagnostic precision, generating detailed reasoning chains before producing final predictions, thereby enhancing transparency and clinical trust\. Generally, there are two main directions in applying the DeepSeek\-R1 strategy to train these medical Long\-CoT reasoning models:

1\) create high\-quality reasoning paths\.The quality of reasoning paths is paramount, as they determine the model’s ability to move beyond simple pattern recognition to provide transparent, trustworthy, and evidence\-based conclusions\.

Specifically, Open\-Medical\-R1qiu2025open, built on the Open\-R1openr1framework, explores optimal data distillation strategies to construct the reasoning dataset\. Similarly, m1huang2025m1proposes a novel data curation pipeline, applying Difficulty Filtering, Thinking Generation, and Diversity Sampling to select a 1K high\-quality subset from the original Medical QA Pairs\. HuatuoGPT\-o1chen2024huatuogptpioneers the concept of “verifiable medical questions,” integrating search\-based reasoning to construct clinically grounded CoT paths, while MedReasonwu2025medreasontransforms QA pairs into logical paths using structured knowledge graphs, where each path is validated for consistency with clinical logic and evidence\-based medicine\. Furthermore, ClinicalGPT\-R1lan2025clinicalgptcombines 20,000 real\-world records with synthetic data, leveraging state\-of\-the\-art LLMs combined with a long\-chain reasoning prompt strategy to generate both steps and final results\. In parallel, EHR\-R1liao2025ehrenhances the reasoning capabilities of foundational models specifically for electronic health record analysis by integrating chain\-of\-thought fine\-tuning\.

2\) optimize the RL framework\.By optimizing the RL framework, researchers can train models to better generalize across diverse tasks and penalize flawed reasoning, ultimately pushing the model’s capabilities from a knowledge retrieval system toward an expert\-level, decision\-making assistant\.

For instance, Med\-U1zhang2025medproposes a unified RL framework for diverse QA formats, optimizing multi\-objective rewards and penalizing shallow reasoning chains, resulting in strong generalization across out\-of\-distribution tasks\. Similarly, Baichuan\-m1wang2025baichuanleverages a three\-stage alignment strategy: Exploratory Log\-likelihood Optimization \(ELO\), Token\-Level Direct Preference Optimization \(TDPO\), and Proximal Policy Optimization \(PPO\), on top of a massive multilingual corpus and a proprietary medical knowledge base, enabling expert\-level evidence\-based reasoning\.

### 4\.3Search\-Guided Reasoning

Search\-guided reasoning refers to methods where the reasoning process of LLMs is actively directed by a structured search algorithm, such as Monte Carlo Tree Search \(MCTS\)browne2012survey;zhang2025llama\. Instead of greedily following a single reasoning chain, the model explores multiple candidate reasoning trajectories, guided by exploration–exploitation trade\-offs\. The MCTS procedure consists of four phases: selection, expansion, simulation, and backpropagation\. In the selection step, the next actiona∗a^\{\*\}is chosen according to the Upper Confidence Bound \(UCB\) criterion:

a∗=arg⁡maxa⁡Q​\(s,a\)\+c⋅ln⁡N​\(s\)N​\(s,a\),a^\{\*\}=\\arg\\max\_\{a\}\\;Q\(s,a\)\+c\\cdot\\sqrt\{\\frac\{\\ln N\(s\)\}\{N\(s,a\)\}\},\(3\)whereQ​\(s,a\)Q\(s,a\)is the estimated value of taking actionaain statess,N​\(s\)N\(s\)is the number of visits to statess,N​\(s,a\)N\(s,a\)is the number of visits to the\(s,a\)\(s,a\)pair, andccis a tunable exploration parameter\.

Therefore, given an LLMℳ\\mathcal\{M\}, the inputℐ\\mathcal\{I\}, and the current statest=\(r1,r2,…,rt\)s\_\{t\}=\(r\_\{1\},r\_\{2\},\.\.\.,r\_\{t\}\), which records the reasoning paths, the LLM will explore the problem space and generateKKviable reasoning paths \(or actions\):

\{rt\+11,rt\+12,…,rt\+1K\}=ℳ​\(ℐ,st\)\\\{r^\{1\}\_\{t\+1\},r^\{2\}\_\{t\+1\},\.\.\.,r^\{K\}\_\{t\+1\}\\\}=\\mathcal\{M\}\(\\mathcal\{I\},s\_\{t\}\)\(4\)Next, from the set of candidate steps generated by the LLM,rt\+1∗r^\{\*\}\_\{t\+1\}is selected by the UCB criterion:

rt\+1∗=arg⁡maxr∈\{rt\+11,rt\+12,…,rt\+1K\}​Q​\(st,r\)\+c⋅ln⁡N​\(st\)N​\(st,r\),r^\{\*\}\_\{t\+1\}=\\underset\{r\\in\\\{r^\{1\}\_\{t\+1\},r^\{2\}\_\{t\+1\},\.\.\.,r^\{K\}\_\{t\+1\}\\\}\}\{\\arg\\max\}\\;Q\(s\_\{t\},r\)\+c\\cdot\\sqrt\{\\frac\{\\ln N\(s\_\{t\}\)\}\{N\(s\_\{t\},r\)\}\},\(5\)Finally, once the best steprt\+1∗r^\{\*\}\_\{t\+1\}is selected, the state \(or the reasoning paths\) is updated by appending this new step\. This represents the expansion of the reasoning path:

st\+1=\(r1,r2,…,rt,rt\+1∗\)\.s\_\{t\+1\}=\(r\_\{1\},r\_\{2\},\.\.\.,r\_\{t\},r^\{\*\}\_\{t\+1\}\)\.\(6\)
In the context of medical reasoning, search\-guided approaches enable the model to evaluate multiple diagnostic hypotheses, simulate different treatment strategies, and select the most promising reasoning path based on accumulated evidence\. Specifically, MedS3jiang2025medsintroduces a novel reasoning evolution framework using MCTS and Process Reward Models \(PRM\), along with a Vote\-Sum strategy for inference\-time selection, achieving significant improvement over 32B generalist models with a smaller scale\. Similarly, AUTOCTliu2025autoctproposes a multi\-agent framework that applies MCTS to iteratively optimize clinical trial prediction performance\.

### 4\.4Retrieval\-Augmented Reasoning

Retrieval\-Augmented Reasoning combines LLM inference with an external retrieval component to inject up\-to\-date, domain\-specific informationarslan2024survey\. In medical domain, this approach addresses the challenge of outdated pretraining data by dynamically fetching relevant guidelines, research literature, and patient\-specific records during reasoningbora2024systematic\. Retrieval\-Augmented Reasoning can be decomposed into three stages: retrieval, fusion, and reasoning\.

Formally, given an input queryℐ\\mathcal\{I\}and a medical knowledge baseDD, the retrieval stage selects a subset of relevant documents:

R=Retriever​\(ℐ,D\),R=\\text\{Retriever\}\(\\mathcal\{I\},D\),\(7\)whereR=\{d1,d2,…,dk\}R=\\\{d\_\{1\},d\_\{2\},\\ldots,d\_\{k\}\\\}denotes the retrieved documents\.

In the fusion stage, the retrieved documents are encoded and aggregated into a contextual representation:

CD=ℱfusion​\(ℐ,R\)=∑i=1kwi⋅ϕ​\(di\),C\_\{D\}=\\mathcal\{F\}\_\{\\text\{fusion\}\}\(\\mathcal\{I\},R\)=\\sum\_\{i=1\}^\{k\}w\_\{i\}\\cdot\\phi\(d\_\{i\}\),\(8\)whereϕ​\(di\)\\phi\(d\_\{i\}\)is the representation of documentdid\_\{i\}andwiw\_\{i\}is its relevance weight \(e\.g\., based on similarity scores or domain\-specific priors\)\. Finally, the reasoning stage generates the answer by conditioning on both the input query and the fused representation:

A=ℳR​A​G​\(ℐ,CD\)\.A=\\mathcal\{M\}\_\{RAG\}\(\\mathcal\{I\},C\_\{D\}\)\.\(9\)
Beyond textual documents, a medical knowledge graphG=\(V,E\)G=\(V,E\)can also serve as an external knowledge source\. Given a queryℐ\\mathcal\{I\}, the retriever extracts a subgraphGℐ⊆GG\_\{\\mathcal\{I\}\}\\subseteq Gcontaining relevant entities and relations\. The subgraph is encoded into a representation by aggregating over its nodes and edges:

CK​G=ℱgraph​\(Gℐ\)=∑\(u,v\)∈Eℐαu​v⋅ψ​\(u,v\),C\_\{KG\}=\\mathcal\{F\}\_\{\\text\{graph\}\}\(G\_\{\\mathcal\{I\}\}\)=\\sum\_\{\(u,v\)\\in E\_\{\\mathcal\{I\}\}\}\\alpha\_\{uv\}\\cdot\\psi\(u,v\),\(10\)whereψ​\(u,v\)\\psi\(u,v\)encodes the relation between entitiesuuandvv, andαu​v\\alpha\_\{uv\}is an attention weight reflecting the importance of this relation\.

In this case, the reasoning stage generates the final answer based on both the input query and the fused representation of the retrieved subgraph:

A=ℳK​G−R​A​G​\(ℐ,CK​G\)\.A=\\mathcal\{M\}\_\{KG\-RAG\}\(\\mathcal\{I\},C\_\{KG\}\)\.\(11\)
Existing medical RAG frameworks mainly focus on the refinement of the retrieval quality for medical documents\. For instance, MMRAGzhan2025mmragretrieves similar examples and lets LLMs to reason in context for biomedical information extraction, while MedBioLMkim2025medbiolmdesigns a knowledge searching and retrieving stage to acquire relevant knowledge, thereby improving the reasoning abilities of LLMs and the factual accuracy\. Similarly, ClinRaGenniu2024multimodalalso proposes to retrieve disease\-related documents from the knowledge base to combine LLM\-driven reasoning with knowledge augmentation\. Building upon this, Self\-BioRAGjeong2024improvingand SelfRewardRAGhammane2024selfrewardragfurther retrieves domain\-specific documents, and applies self\-reflection on generated responses to refine the retrieval results\. Moreover, Med\-R2lu2025medintroduces the Evidence\-Based Medicine \(EBM\) process to improve the trustworthiness of the retrieval stage, while TAGSwu2025tagsproposes a hierarchical retrieval mechanism that selects similar context examples based on both semantic and rationale\-level\.

Meanwhile, models such as MRD\-RAGchen2025mrdand MedRAGzhao2025medragare trying to incorporate comprehensive knowledge graphs as the knowledge base\. The incorporation of structural knowledge allows these models to leverage the relationships and entities within the graph to provide more nuanced and accurate responses\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x8.png)Figure 8:The comparison of various medical LLM Reasoning paradigms\.
### 4\.5Multimodal Reasoning

Medical decision\-making often involves integrating heterogeneous data modalities such as clinical text and medical imagesboehm2022harnessing\. Multimodal reasoning in LLMs aims to jointly process and reason over such inputs to produce coherent outputs\.

Letℐtext\\mathcal\{I\}\_\{\\text\{text\}\}andℐimg\\mathcal\{I\}\_\{\\text\{img\}\}denote the textual and imaging inputs respectively\. First, modality\-specific encoders are used to obtain their representations:

htext=ϕtext​\(ℐtext\),himg=ϕimg​\(ℐimg\)\.h\_\{\\text\{text\}\}=\\phi\_\{\\text\{text\}\}\(\\mathcal\{I\}\_\{\\text\{text\}\}\),\\quad h\_\{\\text\{img\}\}=\\phi\_\{\\text\{img\}\}\(\\mathcal\{I\}\_\{\\text\{img\}\}\)\.\(12\)These representations are then integrated through a multimodal fusion function:

C=ℱfusion​\(htext,himg\)=∑m∈\{text,img\}αm⋅hm,C=\\mathcal\{F\}\_\{\\text\{fusion\}\}\(h\_\{\\text\{text\}\},h\_\{\\text\{img\}\}\)=\\sum\_\{m\\in\\\{\\text\{text\},\\text\{img\}\\\}\}\\alpha\_\{m\}\\cdot h\_\{m\},\(13\)whereαm\\alpha\_\{m\}denotes the modality\-specific attention weight\. Finally, the multimodal reasoning language model generates the output:

A=ℳm​m​\(ℐtext,ℐimg,C\)\.A=\\mathcal\{M\}\_\{mm\}\(\\mathcal\{I\}\_\{\\text\{text\}\},\\mathcal\{I\}\_\{\\text\{img\}\},C\)\.\(14\)
In multi\-modal medical tasks, several recent works have introduced innovative strategies to enhance the reasoning capabilities of large vision\-language models \(LVLMs\)\. These LVLMs are required to comprehend and perform reasoning on these medical images to provide accurate diagnosis and predictions\.

Specifically, Med\-R1lai2025med, QoQ\-Meddai2025qoq, and Gmai\-vl\-r1su2025gmaiapply the DeepSeek\-R1 strategy in LVLMs, enabling these models to support diverse medical imaging modalities, such as CT, X\-ray, and MRI, across a variety of diagnostic and analytical tasks, aiming to create versatile tools for clinical applications\.

Meanwhile, there are also works focused on specifical modalities\. For example, ChestX\-Reasonerfan2025chestxenhances reasoning by implementing a step\-by\-step verification process, which is particularly effective for radiology foundation models, Similarly, Patho\-R1zhang2025pathofocuses on multimodal pathology reasoning, employing Group Relative Policy Optimization and Decoupled CLIP Sampling across 3\.5M image\-text pairs and 500K CoT samples\.

Besides, some works are trying to improving the logical reasoning and factual grounding or develop more efficient training pipelines for medical LVLMs\. Here, MMed\-RAGxia2024mmedintroduces a domain\-aware retrieval mechanism that retrieves multi\-modal documents during the fine\-tuning stage\. At the same time, Med\-E2mu2025elicitproposes a two\-stage post\-training pipeline that elicits and then enhances multimodal reasoning for medical domains, providing a structured and systematic way to improve model performance in medical domains\.

### 4\.6Agentic Reasoning

Agentic reasoning in medicine extends beyond passive information retrieval by enabling LLMs to actively plan, decide, and interact with external tools or environments during the reasoning processli2024mmedagent;schmidgall2024agentclinic\. Unlike traditional RAG systems that primarily focus on enriching the input with domain\-specific knowledge, agentic frameworks introduce autonomy by allowing LLMs to decompose complex clinical queries, orchestrate multi\-step actions, and verify their outputs\. This paradigm is particularly critical in medical contexts where diagnostic reasoning, treatment planning, and patient\-specific recommendations require both factual correctness and adaptive decision\-making\.

Formally, the process can be decomposed into planning, action, interaction, and verification stages\. Given a medical queryℐ\\mathcal\{I\}, the planning module generates a sequence of reasoning steps:

π=𝒫​\(ℐ\)=\{s1,s2,…,sT\},\\pi=\\mathcal\{P\}\(\\mathcal\{I\}\)=\\\{s\_\{1\},s\_\{2\},\\ldots,s\_\{T\}\\\},\(15\)wherests\_\{t\}denotes thett\-th subtask or reasoning step\. At each step, the agent selects an action based on the current state and available external tools:

at=π​\(st,ℰ\),a\_\{t\}=\\pi\(s\_\{t\},\\mathcal\{E\}\),\(16\)whereℰ\\mathcal\{E\}represents the set of external tools or environments \(e\.g\., medical databases, clinical calculators, electronic health records, or simulators\)\. The environment returns a responseoto\_\{t\}:

ot=Env​\(at,ℰ\),o\_\{t\}=\\text\{Env\}\(a\_\{t\},\\mathcal\{E\}\),\(17\)which is fed back into the reasoning loop\. The trajectory of actions and observationsτ=\{\(a1,o1\),…,\(aT,oT\)\}\\tau=\\\{\(a\_\{1\},o\_\{1\}\),\\ldots,\(a\_\{T\},o\_\{T\}\)\\\}is then synthesized by the agentic model to generate the final answer:

A=ℳA​g​e​n​t​\(ℐ,π,τ\)\.A=\\mathcal\{M\}\_\{Agent\}\(\\mathcal\{I\},\\pi,\\tau\)\.\(18\)Therefore, agentic reasoning not only incorporates external medical knowledge but also performs iterative planning and verification, ensuring that the final answerAAis both contextually relevant and clinically reliable\.

Recent studies in medical AI have focused on integrating agentic reasoning with clinical workflows\. For example, Mmedagentli2024mmedagent, MedRAXfallahpour2025medrax, MedOrch\(by University of Maryland\)he2025medorch, and TxAgentgao2025txagentemploy planning and tool\-use mechanisms to guide LLMs in diagnostic decision\-making, while MedAgent\-Prowang2025medagentproposes a step\-by\-step, evidence\-based agentic reasoning paradigm for medical diagnosis\. EHRAgentshi2024ehragentempowers LLMs with code generation capabilities to solve complex tabular reasoning tasks over electronic health records\. Similarly, SMR\-agentswang2026smrand AURAfathi2025auraspecifically integrate multi\-modal medical images for LLM agents to provide decision support, while HiRMedyang2025treeintroduces RAG into the agent framework for intelligent medical test recommendation\. What’s more, Learning to Be A Doctorzhuang2025learningviews medical LLM agents as graph\-structured systems with heterogeneous functional nodes, capable of iterative refinement through diagnostic feedback\. Furthermore, recent advances emphasize conversational and sequential diagnosis; Tu et al\.tu2025towardsand Nori et al\.nori2025sequentialdemonstrate how LLMs can perform iterative information gathering for diagnosis, while Qiu et al\.qiu2025evolvingpropose evolving diagnostic agents within virtual clinical environments to refine decision\-making strategies\.

Beyond single\-agent reasoning, multi\-agent systems have also been proposed to handle the inherent complexity and specialization of medical tasks\. For instance, ColaCarewang2025colacare, MedAidewei2024medaide, MedOrch\(by Fudan University\)chen2025mediator, DoctorAgent\-RLfeng2025doctoragent, MedAgentstang2023medagents, MMedAgent\-RLxia2025mmedagent, MAGDAbani2024magda, and MultiMedResgu2025proactiveare focused on the framework for multi\-agent collaboration, enabling LLM agents to collaborate with each other to solve complex medical tasks\. DruGagentinoue2025drugagent, MEDCOwei2024medco, and ClinicalAgentyue2024clinicalagentutilize multi\-agent systems to predict drug\-target interaction, medical education, and clinical trial, respectively\. Similarly, MAMzhou2025mamintroduce role\-specialized collaboration into the multi\-agent framework to solve multi\-modal medical diagnosis tasks\. Besides, SeM\-Agentschen2025selfproposes a self\-evolving framework for multi\-agent medical consultation, while Tree\-of\-Reasoningpeng2025treeadopts evidence tree to guide multi\-agent reasoning to solve complex medical diagnosis\.

## 5Evaluation of Reasoning Capabilities

The evaluation of medical reasoning systems presents unique challenges that extend beyond traditional NLP metrics to encompass clinical validity, safety, and practical utility\. To address this, we structure our evaluation framework in strict alignment with the proposed five\-level competency hierarchy, ensuring that assessment protocols are tailored to the distinct cognitive demands of each stratum\. As illustrated in Table[5](https://arxiv.org/html/2607.07761#S5.T5), our quantitative analysis adopts a stratified approach: foundational capabilities \(Levels 1\-2\) are assessed via precision\-based metrics for entity normalization and classification accuracy; diagnostic reasoning \(Level 3\) is evaluated on logical consistency and differential diagnosis coverage; while the advanced clinical utility of Levels 4\-5 necessitates complex, outcome\-oriented metrics focusing on decision support quality, safety in treatment planning, and coherence in multi\-turn dialogues\.

To ensure a rigorous and representative assessment, we curated a diverse set of 18 Large Language Models \(LLMs\) based on three primary selection criteria: domain specialization, parameter scale, and architectural paradigm\.

- •Domain Specialization:We included both state\-of\-the\-artGeneral\-purpose LLMs\(e\.g\., GPT\-4o, DeepSeek\-v3\) and specializedMedical/Clinical LLMs\(e\.g\., BioGPT, HuatuoGPT\)\. This dichotomy allows us to investigate whether general reasoning capabilities can outperform domain\-specific pre\-training in complex clinical tasks\.
- •Model Scale Diversity:The selected models span a wide spectrum of parameter sizes, ranging from lightweight models \(e\.g\., BioGPT\-347M, Med\-Gemma\-4B\) to massive\-scale models \(e\.g\., Kimi\-k2\-1T\)\. This variation enables an analysis of scaling laws within the medical domain and helps identify the trade\-offs between computational efficiency and diagnostic performance\.
- •Architectural and Reasoning Paradigms:Beyond standard dense transformers, our selection incorporates diverse architectures, including Mixture\-of\-Experts \(MoE\) and, crucially, the latest reasoning\-enhanced models \(e\.g\., o1, DeepSeek\-R1\-distilled, HuatuoGPT\-o1\)\. Including these reasoning models is essential for evaluating performance in high\-level tasks \(Levels 3\-5\) that require complex logical chains and multi\-step deduction\.

Models\#ParametersLevel 1Level 2Level 3Level 4Level 5MedicalTerm Exp\.MedicalTerm Norm\.OverallDiagnosisPred\.UrgencyDetectionOverallDiagnosticReasoningOverallQA&DecisionSupportTreatmentOutcome Pred\.Length of StayPred\.OverallMulti\-roundDialogDischargeSummaryOverallGeneral\-purpose LLMsGPT\-oss20B0\.2220\.3720\.2970\.2330\.3290\.2610\.6340\.6340\.7000\.6440\.3430\.5970\.2540\.4250\.330Qwen3235B0\.2600\.3500\.3050\.2000\.6420\.3320\.6270\.6270\.6940\.6160\.3770\.5950\.3190\.4120\.360DeepSeek\-v3671B0\.2420\.3800\.3110\.1950\.5840\.3120\.5720\.5720\.7120\.6760\.4110\.6280\.2350\.3620\.291Kimi\-k21T0\.2660\.3120\.2890\.2720\.6050\.3720\.6370\.6370\.7020\.7200\.3570\.6200\.2350\.3640\.292Mistral7B0\.1840\.2720\.2280\.1720\.4500\.2550\.4710\.4710\.3600\.3840\.0000\.2760\.1160\.1490\.131o1N/A0\.2580\.3700\.3140\.2100\.3100\.2400\.6550\.6550\.7440\.6680\.1560\.5780\.3980\.3600\.381DeepSeek\-R1\-distilled8B0\.1960\.3160\.2560\.1660\.2530\.1920\.5190\.5190\.3300\.4880\.1400\.3220\.2890\.4160\.345QWQ\-32B32B0\.0280\.0500\.0390\.1470\.3580\.2100\.5290\.5290\.3720\.5600\.1000\.3510\.4590\.5840\.514Overall\-0\.2070\.3030\.2550\.1990\.4410\.2720\.5810\.5810\.5770\.5950\.2360\.4960\.2880\.3840\.331Medical/Clinical LLMsBioGPT347M0\.0200\.0140\.0170\.0040\.0000\.0030\.0560\.0560\.1780\.0320\.0160\.1010\.1340\.4160\.259HuatuoGPT7B0\.0460\.2000\.1230\.0550\.3580\.1460\.3140\.3140\.2000\.2360\.0360\.1680\.0540\.2040\.120Med\-Gemma4B0\.1560\.4080\.2820\.1840\.7130\.3420\.4940\.4940\.5200\.4760\.0480\.3910\.1020\.1880\.140MedicalGPT8B0\.1380\.2920\.2150\.1790\.5080\.2770\.4590\.4590\.3880\.2520\.0880\.2790\.0950\.1900\.137Baichuan\-m114B0\.2240\.3720\.2980\.2900\.2660\.2710\.4920\.4920\.6360\.6880\.0960\.5140\.2490\.3690\.302HuatuoGPT\-o18B0\.1520\.3000\.2260\.1460\.2920\.1900\.5400\.5400\.3960\.3520\.0080\.2880\.2260\.4980\.346QoQ\-Med7B0\.2220\.3440\.2830\.6050\.0880\.1910\.6020\.6020\.5300\.6600\.0960\.4540\.2130\.3210\.261ClinicalGPT\-r17B0\.1920\.2600\.2260\.5750\.2770\.3370\.6400\.6400\.3120\.8000\.1040\.3820\.1650\.2010\.181OpenMedical\-R112B0\.2140\.3520\.2830\.3450\.3690\.3640\.4760\.4760\.5580\.4400\.0930\.4520\.2710\.3300\.297Overall\-0\.1520\.2820\.2170\.2650\.3190\.2360\.4530\.4530\.4130\.4370\.0650\.3370\.1680\.3020\.227

Table 5:Performance of 18 medical LLMs across five levels of tasks\. The best results in each group arebolded, and the second\-best results areunderlined\. We performed independent t\-tests between general and medical LLMs on the overall scores for each level\. The resulting p\-values are: Level 1 \(p=0\.412p=0\.412\), Level 2 \(p=0\.445p=0\.445\), Level 3 \(p=0\.072p=0\.072\), Level 4 \(p=0\.038p=0\.038\), and Level 5 \(p=0\.041p=0\.041\)\.### 5\.1General Evaluation Framework

Evaluating the reasoning capabilities of Large Language Models \(LLMs\) in medicine requires systematic approaches that capture both output accuracy and reasoning quality\. The evaluation process typically encompasses multiple key components including task formulation, model inference, response analysis, performance aggregation, and clinical validationqiu2025quantifying\. Task formulation structures clinical scenarios into standardized formats, incorporating patient demographics, medical history, symptoms, and test results\. During model inference, systems are prompted to provide explicit reasoning chains alongside final outputs, ensuring transparency in decision\-making processes\. Response analysis examines multiple dimensions including factual accuracy, reasoning completeness, logical consistency, and appropriate handling of uncertaintyreddy2023evaluating\.

The evolution of medical benchmarks has progressed from simple multiple\-choice questions to more complex reasoning\-focused evaluations\. The MMLU\-Pro benchmark represents a significant advancement, expanding answer options from four to ten choices while eliminating trivial questions, thereby reducing random guessing likelihood and increasing discriminative powerwang2024mmlu\. Similarly, MedHELMbedi2025medhelmintroduces a holistic evaluation framework covering diverse medical tasks to assess the general utility of LLMs in healthcare\. Studies utilizing these advanced benchmarks have demonstrated that reasoning models achieve impressive accuracy rates, with DeepSeek\-R1 reaching 95\.1% on complex medical scenarios after expert reconciliationguo2025deepseek\. These models exhibit transparency through chain\-of\-thought reasoning, allowing evaluators to examine logical steps underlying each decision\. However, challenges persist in ensuring comprehensive evaluation beyond simple accuracy metrics, as medical decision\-making requires consideration of timing, context, and real\-world clinical applicabilitygriot2025large;jin2021disease\. Specifically, recent frameworks by Hager et al\.hager2024evaluationand Johri et al\.johri2025evaluationemphasize the need for evaluating safety in clinical decision\-making and the quality of patient interaction tasks beyond pure diagnostic accuracy\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x9.png)Figure 9:The performance comparison of six SOTA methods on our proposed five\-level medical reasoning dataset\.
### 5\.2Fundamental Evaluation Principles

#### 5\.2\.1Clinical Validity and Accuracy

The evaluation of medical reasoning systems must prioritize clinical validity and factual accuracy above all other considerations\. Unlike many other AI applications where errors may be inconvenient or embarrassing, errors in medical reasoning can have direct consequences for patient safety and clinical outcomes\. Factual consistency evaluationtang2023aligninginvolves verifying that system outputs align with established medical knowledge, clinical guidelines, and evidence\-based practices\. This evaluation requires comprehensive medical knowledge bases and expert validation processes to ensure that recommendations are medically sound\. Clinical accuracy assessmenthendrycks2020measuringexamines whether system conclusions and recommendations align with expert clinical judgment across diverse patient scenarios\. This evaluation typically involves comparison with expert clinician assessments and analysis of agreement rates across different clinical contexts\. Diagnostic accuracy evaluationwaqas2025reasoningfocuses specifically on the correctness of diagnostic conclusions, including assessment of sensitivity, specificity, positive predictive value, and negative predictive value across different disease categories and patient populations\. Treatment recommendation accuracybaig2024accuracyexamines whether therapeutic suggestions are appropriate, safe, and evidence\-based for specific patient contexts\. This evaluation must consider not only the appropriateness of individual recommendations but also their integration into comprehensive treatment plans\.

#### 5\.2\.2Reasoning Process Quality

Beyond accuracy of final outputs, the evaluation of medical reasoning systems must assess the quality and completeness of the reasoning processes that lead to clinical conclusions\. This evaluation is crucial for building trust, enabling error detection, and supporting clinical education and decision\-making\. Reasoning completeness assessmenthirosawa2023diagnosticexamines whether systems provide comprehensive reasoning chains that include all relevant clinical considerations\. Incomplete reasoning can lead to missed diagnoses, inappropriate treatments, or failure to consider important risk factors\. Logical consistency evaluationloi2024selfverifies that reasoning chains are internally consistent and free from contradictions\. Medical reasoning often involves complex relationships between multiple clinical factors, and inconsistencies can indicate fundamental flaws in reasoning processes\. Evidence integration assessmentshool2025systematicexamines how effectively systems incorporate and synthesize multiple sources of clinical evidence, including patient data, medical literature, and clinical guidelines\. Effective evidence integration is crucial for comprehensive clinical reasoning\. Uncertainty handling evaluationsavage2024largeassesses how well systems recognize and communicate uncertainty in clinical situations\. Medical practice involves significant uncertainty, and systems must be able to acknowledge limitations in their knowledge or confidence\.

### 5\.3Evaluation Methodologies

#### 5\.3\.1Human Expert Evaluation

Human expert evaluation remains the gold standard for assessing medical reasoning systems, providing clinical validation that cannot be achieved through automated metrics aloneayers2023comparing;johnson2023assessing\. However, expert evaluation is expensive, time\-consuming, and subject to inter\-rater variability, requiring careful design and implementation\. Clinical expert panelswatson2024euprovide comprehensive evaluation of system performance across diverse clinical scenarios\. These panels typically include clinicians from relevant specialties who evaluate system outputs for accuracy, appropriateness, and clinical utility\. Structured evaluation protocolsseo2024evaluationensure consistent and comprehensive assessment across different evaluators and clinical scenarios\. These protocols typically include standardized evaluation criteria, rating scales, and guidelines for handling disagreements between evaluators\. Inter\-rater reliability assessmentchow2016interexamines the consistency of evaluations across different expert evaluators\. High inter\-rater reliability indicates that evaluation results are robust and not dependent on individual evaluator preferences or biases\. Blind evaluationbuhr2023chatgptprocedures prevent evaluator bias by concealing the source of recommendations being evaluated\. This approach enables fair comparison between different systems and between AI systems and human experts\.

#### 5\.3\.2Automated Evaluation Metrics

While human evaluation provides the most clinically relevant assessment, automated metrics enable large\-scale evaluation and continuous monitoring of system performancelee2024analyzing\. These metrics must be carefully designed to capture clinically relevant aspects of performance\. Accuracy\-based metrics, including precision, recall, F1\-score, and area under the ROC curve, provide quantitative assessment of system performance on classification and prediction tasks\. These metrics are particularly useful for diagnostic reasoning and risk prediction applications\. Clinical validity metricsalaa2025medicalassess whether system outputs conform to established medical knowledge and guidelines\. These metrics can be partially automated through comparison with structured medical knowledge bases and clinical decision rules\. Reasoning chain evaluationnachane2024fewmetrics assess the quality of step\-by\-step reasoning processes, including measures of logical consistency, completeness, and evidence integration\. These metrics require sophisticated natural language processing techniques to analyze reasoning quality\. Safety and harm prevention metricsgilson2023does;kung2023performanceidentify potentially dangerous or inappropriate recommendations that could lead to patient harm\. These metrics are crucial for ensuring that systems meet safety requirements for clinical deployment\.

### 5\.4Five\-Level Medical Reasoning Benchmark

To systematically evaluate LLM medical reasoning capabilities according to our proposed competency hierarchy, we have constructed a benchmark dataset that spans all five levels of medical reasoning competence\. This benchmark provides structured assessment aligned with both theoretical frameworks and practical clinical needs\.

#### 5\.4\.1Dataset Composition and Statistics

##### Data Construction and Preprocessing\.

The benchmark dataset comprises 5,000 carefully selected samples, balanced with 1,000 samples for each competency level\. Samples are aggregated from multiple source datasets to ensure diversity and prevent overfitting to specific data distributions\. Prior to inclusion, a rigorous data cleaning pipeline was applied\. This involved: \(1\) De\-identification, ensuring all patient information was anonymized in compliance with HIPAA/GDPR standards; \(2\) Format Standardization, unifying heterogeneous inputs \(e\.g\., SOAP notes, discharge summaries\) into a consistent structured format; and \(3\) Quality Filtering, where incomplete or ambiguous cases were removed via rule\-based heuristics\. The random selection process within each level ensures comprehensive representation across medical specialties \(internal medicine, surgery, pediatrics\), task formats, and patient demographics\.

##### Annotation Procedures\.

To construct the benchmark inputs, we adopted a model\-assisted pipeline to ensure scalability while maintaining clinical relevance\. Specifically, we utilized Large Language Models \(LLMs\) to extract key clinical features and context from existing source benchmarks, restructuring them into a standardized input format\. The diagnostic labels or clinical answers were directly inherited from the original ground truth of the source datasets to preserve the expert\-validated accuracy\. This process ensures that while the input presentation is standardized and enriched by the LLM, the core medical correctness relies on established gold\-standard data\.

##### Inter\-Annotator Agreement and Quality Control\.

Given the semi\-automated nature of the dataset construction, human effort was primarily directed toward quality assurance and validation rather than de novo annotation\. To quantify the reliability of the LLM\-generated inputs and their alignment with the ground truth labels, we implemented a rigorous review protocol\. A subset of 20% of the data was subjected to a double\-blind review, where two independent clinicians assessed the samples for factual consistency, input clarity, and label alignment without access to each other’s evaluations\. The Inter\-Annotator Agreement \(IAA\) on the acceptance of these samples was calculated using Cohen’s Kappa, yielding a score ofκ=0\.88\\kappa=0\.88\. This high degree of concordance confirms the robustness of our pipeline, verifying that the automated reconstruction preserved clinical integrity without introducing hallucinations or semantic distortions\.

#### 5\.4\.2Experimental Analysis on Reasoning Capabilities

We analyze performance along our five\-level competency hierarchy, contrasting general\-purpose LLMs with medical/clinical LLMs and highlighting where specific models excel\.

##### Level 1: Medical term understanding

For Term Expansion, the best model isKimi\-k2, narrowly ahead ofo1; among medical LLMs,Baichuan\-m1leads its cohort but still trails the top general models\. For Term Normalization, the medicalMed\-Gemmais strongest, edgingDeepSeek\-v3\. Overall, general LLMs average higher on Level 1 expansion \(Overall: 0\.207 vs\. 0\.152\), reflecting broad lexical coverage; however, targeted medical pretraining can surpass general models on normalization \(domain\-specific ontology alignment\)\.

##### Level 2: Clinical signal recognition

Medical\-specialist models dominate Diagnosis Prediction\. TheQoQ\-MedandClinicalGPT\-R1clearly outperform all general models\. Conversely, Urgency Detection is led by the lighter medicalMed\-Gemmabut withQwen3close behind\. The General cohort averages higher on this task, and these results suggest domain adaptation is crucial for mapping symptoms to likely diagnoses, whereas urgency cues often benefit from broad world and linguistic priors captured by frontier general LLMs\.

##### Level 3: Diagnostic reasoning

ClinicalGPT\-R1attains the top score\. Despite this medical win at the head, the cohort average favors general models, implying that strong stepwise reasoning remains more consistent in general LLMs, while select medical models can peak when their pretraining aligns with diagnostic COT\.

##### Level 4: Decision support and structured predictions

For QA & Decision Support,o1is best, withDeepSeek\-v3second\. For Treatment Outcome Prediction,ClinicalGPT\-r1leads decisively, withKimi\-k2next\. For Length\-of\-Stay \(LOS\) prediction,DeepSeek\-v3is best\. General models strongly outperform medical models on average in all three Level 4 tasks\. Notably, LOS is challenging for all systems \(low absolute numbers\), reflecting difficulties in learning calibrated, structured temporal outcomes from text alone\.

##### Level 5: Clinical signal recognition

QWQ\-32Bexcels at Multi\-round Dialog andDischarge Summary, witho1andHuatuoGPT\-o1as runners\-up\. Despite some strong medical entrants \(e\.g\., HuatuoGPT\-o1 for summarization\), general models hold the advantage on average, likely due to their richer conversational dynamics and broader stylistic control\. Despite weak scores elsewhere, an interesting outlier isBioGPT, which shows comparatively solid discharge summarization, indicating the value of narrow biomedical language modeling for document\-style generation\.

Taken together, these results suggest a practical division of labor\. When tasks require precise clinical priors and particularly diagnosis\-centric prediction\. Domain\-specialized models provide clear gains\. When tasks demand broad reasoning, instruction adherence, or extended discourse, high\-capacity general LLMs remain stronger on average\. Parameter scale alone does not explain the frontier\. Instruction tuning quality and domain\-constrained curricula are at least as important, and structured clinical predictions \(e\.g\., LOS\) expose current limits in temporal reasoning and calibration\. A system\-level implication is to adopt a routing or mixture\-of\-experts strategy that dispatches diagnosis\-heavy queries to medical models \(e\.g\., QoQ\-Med, ClinicalGPT\-r1\) while leveraging general models \(e\.g\., o1, DeepSeek\-v3, Kimi\-k2, Qwen3, QWQ\-32B\) for decision support, dialogue, and summarization\.

Level TaskModelModel OutputFailure ModeLevel 1TermNormalizationo1*“Standardized term: Postmenopausal Bleeding”*\(Ground truth:*“Vaginal Haemorrhage”*\)Ontology MisalignmenBioGPT*“Standardized term: Chronic Abdominal Pain”*\(Ground truth:*“Back Pain”*\)Ontology MisalignmenLevel 2DiagnosisPredictiono1*“Final diagnosis: Pneumonia”*\(Case labeled as*Healthy*\)Brittle DiagnosisBioGPT*“Final diagnosis: Acute bronchitis”*\(Case labeled as*Pulmonary Embolism*\)Brittle DiagnosisLevel 3DiagnosticReasoningo1*“Positive”*\(Cardiovascular disease diagnosis despite normal blood pressure, normalcholesterol and glucose, no smoking or alcohol use, and an active lifestyle\)Brittle DiagnosisBioGPT*“Negative”*\(Cardiovascular disease diagnosis despite positive ground truth withage\-related risk and active smoking history\)Brittle DiagnosisLevel 4LOSPredictiono1*“Predicted LOS: 4 days”*\(valid JSON format and fluent rationale, but potential temporal under\-specification\)Temporal Under\-SpecificationBioGPT*“If the patient was male, he was female, and if the patient was on dialysis, he was male\.”*\(logical inconsistency and loss of patient attribute coherence\)Logical InconsistencyLevel 5DischargeSummaryo1*“Upon discharge, the patient’s condition was stable …”*\(coherent narrative flow with risk of inferred or reordered events\)Narrative Over\-InterpretationBioGPT*“Patient Condition: … History of Present Illness: … Discharge Instructions: …”*\(template\-style listing with limited narrative integration\)Rigid Templating

Table 6:Representative failure cases across Levels 1–5\. Failures at Levels 1–3 are localized, including ontology misalignment in term normalization and brittle diagnostic reasoning under non\-canonical symptom patterns\. In contrast, Levels 4 and 5 expose more fundamental limitations related to execution fidelity, temporal grounding, and discourse\-level control\.
##### Failure case analysis and qualitative findings

Failure case analysis reveals systematic and level\-dependent limitations that are not fully captured by aggregate metrics\. At Levels 1–3, observed failures are comparatively localized: general LLMs occasionally misalign surface medical terms with formal ontologies during normalization, while medical LLMs exhibit brittle diagnostic reasoning when symptom patterns deviate from canonical clinical templates\. These errors, however, remain bounded and do not dominate overall performance trends\.

In contrast, Level 4 structured prediction tasks, particularly Length\-of\-Stay \(LOS\), expose pronounced divergences between model families\. Medical LLMs frequently fail at the execution level\. For example, BioGPT occasionally generates logically inconsistent text\. These behaviors indicate difficulties in aligning clinical safety constraints with strict output specifications\. By contrast, general\-purpose LLMs such aso1reliably adhere to required JSON formats and generate fluent reasoning traces, even when their LOS predictions are inaccurate\. This execution robustness largely explains the statistically significant performance advantage of general models at Level 4: they function as competent task executors, whereas medical LLMs often fail before the correctness of the prediction can be meaningfully evaluated\.

Level 5 discharge summarization further highlights qualitative differences in generative behavior\. General LLMs produce summaries with strong narrative flow, organizing clinical events into coherent, temporally ordered stories \(e\.g\., “Upon discharge, the patient…”\), closely resembling real\-world clinical documentation\. In contrast, medical LLMs such as BioGPT favor rigid, list\-like structures \(e\.g\., “Patient Condition:”, “History:”\) that preserve factual elements but lack integrative writing and discourse\-level synthesis\. While this narrative capacity contributes to higher average scores for general models, it also introduces risks of over\-interpretation, including event reordering or inferred causal relations not explicitly supported by the source text\.

Taken together, these failure cases suggest that higher\-level clinical tasks are constrained not only by medical knowledge coverage, but by execution fidelity, format adherence, and discourse\-level control\. The results reinforce a system\-level design principle: domain\-specialized medical models are best suited for diagnosis\-centric and safety\-critical subtasks, while general LLMs are more reliable for structured execution, dialogue management, and narrative clinical documentation\.

![Refer to caption](https://arxiv.org/html/2607.07761v1/x10.png)Figure 10:Data and knowledge challenges in medical reasoning with large language models, and a comparison between correlation\-based model diagnosis and causal, evidence\-grounded clinical reasoning in chest pain cases\.

## 6Challenges and Limitations

Despite significant advances in medical reasoning systems, numerous challenges and limitations continue to constrain their development, deployment, and clinical impactyu2018artificial;al2023review;secinaro2021role\. Understanding these challenges is crucial for directing future research efforts and setting realistic expectations for clinical implementation\.

### 6\.1Data and Knowledge Challenges

#### 6\.1\.1Data Scarcity and Quality Issues

The effective development of medical reasoning systems is fundamentally constrained by the availability of large\-scale, high\-quality, and annotated training data\. As shown in Fig\.[10](https://arxiv.org/html/2607.07761#S5.F10), medical data faces severe challenges in terms of annotation complexity, privacy protection, reliance on domain expertise, timeliness, and data imbalanceli2021critical\. High\-quality medical data not only requires professionals to spend substantial time and effort on annotation, but also suffers from issues such as privacy and regulatory restrictions, severe shortage of rare disease samples, and rapid medical knowledge obsolescence\. More seriously, differences in judgment among experts often make it difficult to form a unified “gold standard”murdoch2021privacy;thapa2021precision\. In some complex reasoning tasks, such as diagnostic reasoning or treatment planning, annotation requires not only the correct answers but also detailed reasoning chains to reflect the expert’s thought process, which is costly and time\-consuming\. In addition, certain privacy and regulatory restrictions \(such as the HIPAA regulatory process in the United States, shown in Fig\.[10](https://arxiv.org/html/2607.07761#S5.F10)\) further reduce the size and diversity of datasets\. What’s more, data imbalance is particularly prominent in medical applications; as shown in Fig\.[10](https://arxiv.org/html/2607.07761#S5.F10), rare diseases and uncommon cases are severely underrepresented in training data, leading to poor performance of medical reasoning systems in these clinically important scenarios\. In addition, with the rapid evolution of medical knowledge and best practices, outdated medical data may cause reasoning systems to recommend obsolete therapies or fail to incorporate the latest reasoning guidelines\. Furthermore, in complex clinical scenarios, inter\-annotator agreement \(IAA\) can be very low, making it difficult to establish the “gold standard” required for training and evaluationzajkac2023ground\. Therefore, the development of medical reasoning systems is severely constrained by data scarcity and quality issues\.

#### 6\.1\.2Knowledge Representation and Integration

Medical reasoning needs to combine different types of knowledge, including medical facts, clinical guidelines, research literature, and clinical experience\. However, there remain significant challenges in how to represent and integrate this knowledgeriano2019ten\. For example, existing medical knowledge bases \(such as ”UMLS” and ”SNOMED CT”\), although broad in coverage, are insufficient in capturing the finer semantics and specific contexts required in clinical reasoningfung2005integrating\. Moreover, clinical guidelines are usually written in natural language, often containing ambiguity, exceptions, or even conflicting recommendations, which makes transforming them into machine\-usable formats both complex and error\-prone\. In addition, the scale of medical literature is vast and continuously growing, and current systems remain limited in their ability to filter, evaluate, and integrate evidence\. Meanwhile, clinical experience often depends on specific contexts and carries implicit characteristics, making it even harder to formalize and utilize\. Overall, medical reasoning not only relies on factual knowledge but also requires the integration of guidelines, literature, and experience, and how to effectively represent and combine these diverse sources of knowledge remains a central challenge for medical reasoning systems\.

### 6\.2Model Limitations and Technical Challenges

#### 6\.2\.1Hallucination and Factual Accuracy

One prominent risk of large language models in medical applications is that they may generate content that appears reasonable but is in fact incorrect, a phenomenon known as hallucinationji2023survey\. In medical scenarios, such errors may have serious consequences for patient safety and clinical decision\-making\. This problem is mainly reflected in four aspects: First, the erroneous content of model hallucinations can take many forms\. Hallucinations may appear as incorrect drug dosages, fabricated diseases, inappropriate treatment recommendations, or even invented research citations that do not existagarwal2024medhalu;kim2025medical\. Such content is often highly deceptive and can easily mislead non\-professional users\. Second, the confidence calibration of the model is insufficient\. Medical reasoning systems often perform poorly in judging the reliability of their own answers\. As shown in Fig\.[10](https://arxiv.org/html/2607.07761#S5.F10), they may exhibit excessive confidence in incorrect answers, which leads users to over\-rely on them and increases risk\. Third, automated fact\-checking is extremely difficult\. Medical knowledge is not only complex but also highly context\-dependent\. Automated fact\-checking must simultaneously determine whether medical facts are correct, clinically applicable, and valid in specific contexts, which makes the verification process very challengingsarrouti2021evidence\. Finally, the issue of insufficient traceability and lack of evidence support also deserves attention\. Large language models often generate content without authoritative sources, making it difficult for clinicians to verify the information and to build sufficient trust in the model’s output\. In summary, the problems of hallucination and factual accuracy constitute one of the core challenges of medical reasoning models and are key obstacles limiting their safe clinical application\.

#### 6\.2\.2Reasoning Depth and Complexity

Existing medical reasoning systems often remain at the level of shallow pattern matching and lack a deep understanding of medical principles and causal relationshipsholzinger2019causability\. This deficiency is mainly reflected in three aspects: First, the causal reasoning ability of diagnostic models is limited\. Models often only identify correlations in the data but fail to grasp the underlying causal mechanisms, which may lead to incorrect recommendations based on spurious correlations\. Second, multi\-step reasoning chains are unstable\. In complex tasks that require multi\-step logical chains, systems often lose coherence or exhibit logical breaks when integrating information from multiple sources\. In addition, uncertainty handling and counterfactual reasoning are also highly insufficient\. Since medical reasoning involves multiple uncertainties, systems still face difficulties in quantifying, propagating, and interpreting these uncertainties\. At the same time, existing models lack counterfactual reasoning ability \(i\.e\., considering alternative scenarios and their effects\), which is particularly critical in treatment planning and risk assessmentyang2023counterfactual;prosperi2020causal\. A typical case is the differential diagnosis of chest pain, as shown in Fig\.[10](https://arxiv.org/html/2607.07761#S5.F10)\. Angina pectoris and gastroesophageal reflux overlap greatly in symptoms \(such as retrosternal pressure\-like discomfort\)\. Data\-driven correlation models may directly map “chest pain and nocturnal or postprandial discomfort” to a single diagnosis, resulting in hallucination\-like misjudgment\. In contrast, clinicians analyze along the causal chain the inducing and relieving factors \(for example, exertion\-induced, relieved by rest or nitrates indicating myocardial ischemia; aggravated when lying down or after meals and accompanied by acid reflux indicating gastroesophageal reflux\), and then combine evidence such as ECG, troponin, or proton pump inhibitor trial to verify and produce a more accurate diagnosis\. This contrast shows that systems lacking pathophysiological causal modeling and evidence traceability are easily misled by superficial correlations and cannot ensure reliability in critical clinical scenarios\.

#### 6\.2\.3Scalability and Computational Requirements

Complex medical reasoning systems often require enormous computational power, which is in natural conflict with the clinical demand for efficient applicationswahl2018artificial\. In some routine scenarios, slower inference speed may only reduce the user experience, but as shown in Fig\.[10](https://arxiv.org/html/2607.07761#S5.F10), in emergency and critical care settings where real\-time response is required, excessively long inference times can directly affect patient safety\. Furthermore, medical resource limitations make the problem even more prominent\. Many hospitals lack sufficient computing infrastructure and budget, while resource\-limited regions \(such as rural or low\-resource hospitals\), although in greatest need of AI support, are least able to afford the deployment and maintenance costs of high\-computation systems\. Beyond this, sustainability issues cannot be ignored\. Large\-scale model inference not only consumes enormous energy but also brings long\-term environmental burdens and operational pressuresstrubell2020energy\. Therefore, from conflicts with real\-time requirements, to resource insufficiency, and further to energy consumption and sustainability, the computational bottleneck has gradually become the most severe obstacle to the clinical implementation of medical reasoning systems\.

## 7Recent Advances and Future Directions

The field of medical reasoning with large language models is rapidly evolving, with significant advances occurring across multiple dimensions including technical capabilities, clinical applications, and regulatory frameworks\. This section examines recent breakthroughs and identifies promising directions for future research and development\.

### 7\.1Advanced Reasoning Architectures

Recent developments in reasoning architectures have significantly enhanced the capabilities of medical reasoning systems, moving beyond simple question\-answering toward more sophisticated clinical reasoning processes\. Long\-form chain\-of\-thought reasoningchen2025towardshas emerged as a particularly promising approach for medical applications, enabling systems to engage in extended reasoning processes that more closely mirror expert clinical thinking\. Models like OpenAI’s o1 and DeepSeek\-R1 have demonstrated that extended reasoning can significantly improve performance on complex reasoning tasks, and early applications to medical domains show similar promise\. The integration of tool\-augmented reasoningma2024sciagentwith medical knowledge baseshe2025medorchand clinical decision support tools has enabled more accurate reasoning\. These systems can access up\-to\-date medical information, perform calculations, and integrate multiple sources of evidence during the reasoning process\. Self\-reflection and error correction mechanismsji2023towardshave been developed to enable systems to review and refine their reasoning processes\. These capabilities are particularly important in medical contexts where accuracy is paramount and errors can have serious consequences\. Multi\-agent reasoning systemsvicari2003multithat combine multiple specialized models or reasoning approaches have shown promise for complex medical scenarios that require diverse types of expertise\. These systems can leverage the strengths of different models while mitigating individual weaknesses\.

### 7\.2Multimodal Integration Advances

The integration of multiple data modalities has seen significant advances, enabling more comprehensive and accurate medical reasoning\. Vision\-language integration has reached new levels of sophistication, with models like GPT\-4Vyang2023dawn, DeepMedix\-R1lin2025foundation, and Med\-Geminisaab2024capabilitiesdemonstrating impressive capabilities in analyzing medical images and integrating visual information with clinical text\. These advances enable reasoning about radiology images, pathology slides, and clinical photographs in conjunction with textual clinical information\. Time\-series integration capabilitiesli2014physiologicalhave improved significantly, enabling better reasoning about physiological monitoring data, laboratory trends, and disease progression patterns\. These capabilities are particularly important for intensive care and chronic disease management applications\. Structured data integrationwu2025medreasonhas advanced to enable seamless reasoning across electronic health record data, laboratory results, and clinical documentation\. These advances enable more comprehensive clinical reasoning that considers all available patient information\. Genomic data integrationioannidis2018evidencerepresents an emerging frontier, with early systems demonstrating the ability to integrate genetic information with clinical data for personalized medicine applications\.

### 7\.3Safety and Reliability Improvements

Significant advances have been made in improving the safety and reliability of medical reasoning systems, addressing critical concerns about clinical deployment\. Uncertainty quantification methodsatf2025challengehave become more sophisticated, enabling systems to provide more accurate estimates of their confidence in different recommendations\. These improvements help clinicians understand when system outputs should be trusted and when additional verification is needed\. Hallucination detection and mitigation techniqueskim2025medicalhave been developed specifically for medical applications, using domain\-specific knowledge bases and fact\-checking mechanisms to identify and prevent factually incorrect outputs\. Adversarial robustnesschen2025careshas been improved through specialized training techniques that make systems more resistant to inputs designed to elicit incorrect or harmful responses\. Safety guardrails and content filtering mechanismsyang2024ensuringhave been developed to prevent systems from generating potentially harmful recommendations, such as dangerous drug combinations or inappropriate treatment suggestions\.

## 8Conclusion

This survey has addressed the growing challenges in medical decision\-making, where clinicians must process large volumes of data while managing uncertainty and complex trade\-offs\. Through our dual\-view framework linking clinical competencies with reasoning types, we have clarified what kinds of reasoning are actually needed in clinical practice and how current LLM capabilities map to these specific medical needs across the care pathway\.

Our five\-level competency framework reveals that current models perform well on knowledge recognition and classification tasks but face important challenges with higher\-level reasoning requiring integration, personalization, and dynamic interaction\. The evaluation of 18 state\-of\-the\-art models on our benchmark demonstrates a clear performance pattern: medical specialist models excel in diagnosis\-centric tasks while general models show stronger performance in decision support, dialogue, and summarization scenarios\. Notably, temporal predictions such as length\-of\-stay remain difficult for all systems, and model size alone does not explain performance differences\. Instruction quality, domain\-specific data, and reasoning training appear equally important\.

These findings point toward practical deployment strategies that route diagnosis\-heavy queries to specialist models while using general models for supportive tasks\. However, key limitations remain in areas such as uncertainty handling, grounding to trusted medical sources, and integration into real clinical workflows\. Moving forward, the field requires better evaluation frameworks that assess both accuracy and reasoning quality, improved training approaches that combine domain expertise with robust reasoning capabilities, and careful attention to safety and workflow integration\.

## Statements and Declarations

The authors have no relevant financial or non\-financial interests to disclose\.

## Acknowledgments

This work was supported by the Guangdong Provincial Fund for Basic and Applied Basic Research—Regional Joint Fund Project \(Key Project\) \(No\. 2023B1515120078\), the National Natural Science Foundation of China \(No\. 62476097\), the Science and Technology Planning Project of Guangdong Province \(No\. 2025B0101120003\), the Guangdong Provincial Natural Science Foundation for Outstanding Youth Team Project \(No\. 2024B1515040010\), the Fundamental Research Funds for the Central Universities, South China University of Technology \(No\. x2rjD2250190\), the National Natural Science Foundation of China \(No\. 62372314\), the 1\.3\.5 project for disciplines of excellence from West China Hospital of Sichuan University \(No\. 2023HXFH044\), and the Hong Kong Research Grants Council under the Theme\-based Research Scheme \(Project No\. T22\-501/23\-R\)\. The experimental part of this work was supported by The Centre for Large AI Models \(CLAIM\) of The Hong Kong Polytechnic University\.

## References

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/pengqi.jpg)Qi Pengis currently a dual Ph\.D\. student at Hong Kong Polytechnic University and South China University of Technology, supervised by Prof\. Qing Li and Prof\. Yi Cai\. He received both his Master’s degree and Bachelor’s degree in South China University of Technology in 2022 and 2019, respectively\. His research interests include large language models, multi\-agent systems, and intelligent healthcare\. His research works have been published as the first author in ACM MM, IEEE TMI, and KBS\.

E\-mail: qi0125\.peng@connect\.polyu\.hk

ORCID iD: 0000\-0001\-7747\-3293

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/jiatongli.jpg)Jiatong Liis currently a PhD candidate of the Department of Computing \(COMP\), The Hong Kong Polytechnic University \(funded by HKPFS\)\. Before joining the PolyU, he received his Master’s degree of Information Technology \(with Distinction\) from the University of Melbourne\. In 2021, he got his bachelor’s degree in Information Security from Shanghai Jiao Tong University\. His interest lies in Natural Language Processing, Drug Discovery, Medicine, and healthcare\. He has published innovative works in top\-tier conferences and journals such as IJCAI, ACL, CVPR, and IEEE TKDE\. For more information, please visit https://phenixace\.github\.io/\.

ORCID iD: 0000\-0001\-7705\-2296\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/siruihuang.jpg)Sirui Huangreceived the MSc degree from Electronic and Electrical Engineering Department, University College London, London, U\.K\., in 2021\. She is currently a Ph\.D\. student in the Computing Department at Hong Kong Polytechnic University, Hong Kong, China, supervised by Prof\. Qng Li and Prof\. Guandong Xu\. Her research interests mainly include recommender systems, large language models and table reasoning\.

ORCID iD: 0000\-0003\-1206\-2260\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/yiyangjiang.jpg)Yiyang Jiangreceived the B\.Sc\. \(Hons\., 1st Class\) degree in Computer Science from The Hong Kong Polytechnic University in 2024\. He is currently a Ph\.D\. student in the Department of Computing at The Hong Kong Polytechnic University, supported by the Hong Kong PhD Fellowship Scheme from the Research Grants Council\. His research interests include vision\-language understanding, natural language processing, large language models, and health computing\.

ORCID iD: 0009\-0007\-1169\-4465\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/kaisonggong.jpg)Kaisong Gongreceived the B\.S\. degree from Tianjin University in 2024\. He is currently a graduate student at The University of Hong Kong and a research assistant at The Hong Kong Polytechnic University\. His research interests include VLLM reasoning and parameter\-efficient fine\-tuning\.

E\-mail: gks70853@connect\.hku\.hk,

ORCID iD: 0009\-0008\-7612\-3177\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/rongerding.jpg)Ronger Dingis an undergraduate student at the Software College of South China University of Technology\. Her research interest include the application of large language models in various fields\.

E\-mail: se30480335ding@mail\.scut\.edu\.cn\.

ORCID iD: 0009\-0005\-6422\-1130\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/shijieye.jpg)Shijie Yeis currently pursuing the B\.Sc\. degree in Mathematics and Statistics at the University of Toronto\. He is preparing to pursue a Ph\.D\. degree in Artificial Intelligence, with research interests focusing on multimodal AI, large language models, deep learning, and their applications in the medical domain\.

ORCID iD: 0009\-0001\-6681\-7190\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/changmengzheng.jpg)Changmeng Zhengis currently a Research Assistant Professor with the Department of Computing, the Hong Kong Polytechnic University, Hong Kong SAR\. He received his Bachelor’s and Master’s degree from South China University of Technology, and the Ph\.D\. degree from the Hong Kong Polytechnic University\. His research work has been published in refereed journals and conferences such as IEEE Transactions on Multimedia, IEEE Transactions on Circuits and Systems for Video Technology, Neural Networks, ACL, EMNLP, COLING, ACM MM\. His research interests are in the areas of multimodal learning and social media analytics, especially the knowledge graph and large language models\.

E\-mail: changmeng\.zheng@polyu\.edu\.hk \(Corresponding Author\)\.

ORCID iD: 0000\-0002\-2945\-8248\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/caiyi.jpg)Yi Cai, professor, serves as Dean of the School of Software Engineering at South China University of Technology\. He holds concurrent appointments as Director of the Ministry of Education’s Key Laboratory of Big Data and Intelligent Robotics and Director of the SCUT\-Kingsoft Office Software Joint Laboratory\. As a Distinguished Member of the China Computer Federation, Professor Cai participates in several technical committees related to natural language processing and database systems\. He holds leadership roles in professional organizations including the Guangdong Big Data Committee and the Standardized Knowledge Graph Alliance\. Professor Cai has published more than 200 research papers in peer\-reviewed journals and conference proceedings, including IEEE Transactions on Knowledge and Data Engineering and IEEE/ACM Transactions on Audio, Speech and Language Processing\. His research output includes 30 granted patents\. He has contributed to the development of technical standards in artificial intelligence through both national and international standardization initiatives\. His research has received recognition including awards from the Geneva International Exhibition of Inventions, Guangdong Province, and the China Computer Federation\. His research team has achieved competitive results in technical evaluations related to big data processing and natural language understanding\.

E\-mail: ycai@scut\.edu\.cn

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/yangxiaobo.jpg)Xiaobo Yang, M\.D\., Ph\.D\., is an Associate Chief Physician in the Department of Surgery at Peking Union Medical College Hospital \(PUMCH\), where he has served since 2013 after receiving his medical doctorate from Peking Union Medical College\. He has been recognized as Outstanding Resident Physician and one of the “Top Ten Attending Physicians” at PUMCH, and has received multiple PUMCH Medical Achievement Awards, as well as the 3rd Prize of the Chinese Medical Science and Technology Award\.

E\-mail: y110403606@126\.com

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/huangjin.png)Jin Huang, M\.D\. PhD, is a Professor and Doctoral Supervisor n Urology at West China Hospital, Sichuan University\. Executive Vice President of West China Hospital; and Director of the Center for Medical\-Engineering Innovation and Translational Research\. He currently serves as Vice Chair of the Medical Engineering Branch of the Chinese Medical Association, Vice President of the China Association of Medical Equipment, among other leadership roles\. He has been selected as Sichuan Provincial Academic and Technological Leader and Tianfu Qingcheng Plan Science and Technology Innovation Leading Talent\. His work focuses on population\-based health checkup cohorts, early\-screening mechanisms for urological comorbidities, and digital\-health translation\. Professor Huang has led 13 national\-level projects, including the National Key R&D Program and the National Natural Science Foundation of China\. He has published over 100 high\-impact papers as first or corresponding author in journals such as BMJ, Nature Mental Health, and the European Journal of Epidemiology\. His research was included in the BMJ “2025 China Hospital Research Impact Report” and has been cited by 55 global health policy and clinical guidelines\.

E\-mail: michael\_huangjin@163\.com\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/xiaoyongwei.jpg)Xiaoyong Weireceived the Ph\.D\. degree in computer science from City University of Hong Kong, China in 2009, and has worked as a postdoctoral fellow in the University of California, Berkeley, USA from December, 2013 to December, 2015\. He has been a professor and the head of Department of Computer Science, Sichuan University, China since 2010\. He is an adjunct professor of Peng Cheng Laboratory, China, and a visiting professor of Department of Computing, Hong Kong Polytechnic University\. He is a senior member of IEEE, and has served as an associate editor of Interdisciplinary Sciences: Computational Life Sciences since 2020, the program Chair of ICMR 2019, ICIMCS 2012, and the technical committee member of over 20 conferences such as ICCV, CVPR, SIGKDD, ACM MM, ICME, and ICIP\. His research interests include multimedia computing, health computing, machine learning and large\-scale data mining\.

E\-mail: x1wei@polyu\.edu\.hk\.

ORCID iD: 0000\-0002\-5706\-5177\.

![[Uncaptioned image]](https://arxiv.org/html/2607.07761v1/author_photos/qingli.jpg)Qing Liis currently a Chair Professor \(Data Science\) and the Head of the Department of Computing, the Hong Kong Polytechnic University\. Formerly, he was the founding Director of the Multimedia software Engineering Research Centre \(MERC\), and a Professor at City University of Hong Kong where he worked in the Department of Computer Science from 1998 to 2018\. Prior to these, he has also taught at the Hong Kong University of Science and Technology and the Australian National University \(Canberra, Australia\)\. Prof\. Li served as a consultant to Microsoft Research Asia \(Beijing, China\), Motorola Global Computing and Telecommunications Division \(Tianjin Regional Operations Center\), and the Division of Information Technology, Commonwealth Scientific and Industrial Research Organization \(CSIRO\) in Australia\. He has been an Adjunct Professor of the University of Science and Technology of China \(USTC\) and the Wuhan University, and a Guest Professor of the Hunan University \(Changsha, China\) where he got his BEng\. degree from the Department of Computer Science in 1982\. He is also a Guest Professor \(Software Technology\) of the Zhejiang University \(Hangzhou, China\) – the leading university of the Zhejiang province where he was born\. Prof\. Li has been actively involved in the research community by serving as an associate editor and reviewer for technical journals, and as an organizer/co\-organizer of numerous international conferences\. Some recent conferences in which he is playing or has played major roles include APWeb\-WAIM’18, ICDM 2018, WISE2017, ICDSC2016, DASFAA2015, U\-Media2014, ER2013, RecSys2013, NDBC2012, ICMR2012, CoopIS2011, WAIM2010, DASFAA2010, APWeb\-WAIM’09, ER’08, WISE’07, ICWL’06, HSI’05, WAIM’04, IDEAS’03,VLDB’02, PAKDD’01, IFIP 2\.6 Working Conference on Database Semantics \(DS\-9\), IDS’00, and WISE’00\. In addition, he served as a programme committee member for over fifty international conferences \(including VLDB, ICDE, WWW, DASFAA, ER, CIKM, CAiSE, CoopIS, and FODO\)\. He is currently a Fellow of IEEE and IET/IEE, a member of ACM\-SIGMOD and IEEE Technical Committee on Data Engineering\. He is the chairperson of the Hong Kong Web Society, and also served/is serving as an executive committee \(EXCO\) member of IEEE\-Hong Kong Computer Chapter and ACM Hong Kong Chapter\. In addition, he serves as a councilor of the Database Society of Chinese Computer Federation \(CCF\), a member of the Big Data Expert Committee of CCF, and is a Steering Committee member of DASFAA, ER, ICWL, UMEDIA, and WISE Society\.

E\-mail: qing\-prof\.li@polyu\.edu\.hk\.

ORCID iD: 0000\-0003\-3370\-471X\.