FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning

# FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning
Source: [https://arxiv.org/html/2607.01440](https://arxiv.org/html/2607.01440)
Zhiyun Zhang1∗\\ast,Liwen Sun1∗\\ast,Xiang Qian2,Chenyan Xiong1,3 1Carnegie Mellon University2Stanford University School of Medicine3Xlue ∗\\astEqual contribution \{zhiyunz, liwens, cx\}@andrew\.cmu\.edu, xqian@stanford\.edu

###### Abstract

Faithful reasoning is essential in medicine, where clinical decisions require transparent justification grounded in reliable evidence\. Current medical LLMs either lack active access to evidence or use retrieved evidence without supervising how it should be appraised and applied during reasoning\. To address this, we formalize evidence\-based medicine principles as process\-level criteria and introduce FaithMed, a framework that combines clinician\-designed, automatically refined rubrics with reinforcement learning using step\-level process reward assignment and advantage grouping\. Across seven medical benchmarks, FaithMed improves over agentic\-search baselines \(\+9% on average\) and outcome\-only RL \(\+5\.8%\), while raising average evidence\-based medicine rubric scores over agentic\-search Qwen3 baselines \(\+15\.5%\)\. This work demonstrates that explicit step\-level supervision can improve both task success and the faithfulness of the reasoning process\. Code is available at[https://github\.com/cxcscmu/FaithMed](https://github.com/cxcscmu/FaithMed)\.

FaithMed: Training LLMs For Faithful Evidence\-Based Medical Reasoning

## 1Introduction

In medicine, reliable clinical decision\-making depends on reasoning that connects patient\-specific information with trustworthy evidence and makes its justification transparent\(Schwartz and Elstein,[2008](https://arxiv.org/html/2607.01440#bib.bib21)\)\. Evidence\-based medical reasoning operationalizes this goal, requiring physicians to synthesize patient history, clinical findings, and medical guidelines to support accurate diagnoses and treatment planning\(Sackett et al\.,[1996](https://arxiv.org/html/2607.01440#bib.bib20)\)\. In real\-world healthcare settings, this reasoning process is often complex, time\-consuming, and vulnerable to incomplete documentation, noisy observations, and inconsistent evidence across longitudinal patient records\(Gruppen,[2017](https://arxiv.org/html/2607.01440#bib.bib8); Temsah et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib29)\)\.

Recent advances in large language models \(LLMs\) have demonstrated strong capabilities in medical question answering \(QA\) and clinical reasoning, motivating the development of automated systems that can assist clinicians in interpreting medical evidence and supporting healthcare decisions\(Jin et al\.,[2019](https://arxiv.org/html/2607.01440#bib.bib13); Singhal et al\.,[2023b](https://arxiv.org/html/2607.01440#bib.bib25); Sun et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib27)\)\. However, ensuring that these systems produce faithful and evidence\-grounded reasoning remains critical for improving reliability, transparency, and patient safety in high\-stakes clinical environments\(Singhal et al\.,[2023a](https://arxiv.org/html/2607.01440#bib.bib24); Saab et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib19)\)\.

![Refer to caption](https://arxiv.org/html/2607.01440v1/x1.png)Figure 1:Comparing unfaithful reasoning with evidence\-based medical reasoning\.To improve factual grounding and attribution, prior work has explored retrieval\-augmented generation \(RAG\), which combines LLMs with external trustworthy medical knowledge sources to generate verifiable responses with supporting citations\(Wang et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib31); Xiong et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib35); Gao et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib7)\)\. While these approaches can improve answer correctness and attribution, the reasoning process itself may still be unfaithful to the supporting evidence\. In particular, models may generate plausible rationales that drift from the patient context, cite irrelevant evidence, or make claims that are not fully supported by the retrieved information, as illustrated in Figure[1](https://arxiv.org/html/2607.01440#S1.F1)\. Although the final answer is correct, the underlying reasoning process can still be flawed and unreliable\.

To address this limitation, we introduce FaithMed, a framework for faithful evidence\-based medical reasoning that trains LLMs to generate evidence\-supported reasoning through rubric\-guided reinforcement learning\. Inspired by real\-world evidence\-based medicine workflows—ask,acquire,appraise,apply, andassess—we collaborate with clinicians to design initial rubrics that evaluate reasoning quality across these five dimensions, then automatically refine them to reduce redundancy and improve discriminativeness\. We then integrate these refined rubrics into a step\-level reinforcement learning framework with rubric\-guided rewards\. Instead of producing unconstrained rationales, FaithMed encourages the model to follow clinically meaningful reasoning by grounding intermediate reasoning steps in supporting evidence and generating accurate answers consistent with the evidence\-based reasoning process\.

Across seven medical benchmarks, FaithMed improves answer accuracy by 9% over agentic\-search methods and by 10\.8% over outcome\-only RL baselines, while achieving a 15\.5% gain in evidence\-based medicine rubric scores\. These gains suggest that faithful process supervision improves not only the reasoning trace but also final task performance\. We further investigate different faithful reasoning injection strategies by comparing episode\-level and step\-level process rewards in reinforcement learning, and show that fine\-grained step\-level rewards derived from individual rubrics achieve superior performance\. Moreover, our analysis reveals that episode\-level process rewards can incorrectly assign credit across step groups, leading to suboptimal optimization\. Finally, clinician\-annotated case studies demonstrate that the reasoning produced by our method aligns more closely with the corresponding evidence\-based medicine dimensions than strong baselines with agentic search\.

Our main contributions can be summarized as follows:

- •We propose a framework for faithful evidence\-based medical reasoning that enables LLMs to generate clinically grounded responses with evidence\-supported reasoning processes\.
- •We develop clinician\-designed and automatically refined rubrics for faithful reasoning evaluation, and use them for reinforcement learning with step\-level process reward assignment and advantage grouping\.
- •We demonstrate that our method improves not only answer accuracy but also reasoning faithfulness, outperforming existing baselines across multiple medical QA benchmarks\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2607.01440v1/x2.png)Figure 2:Method overview\. We first design and refine rubrics for evaluating the reasoning process across five evidence\-based medicine dimensions\. We then use these rubrics to construct step\-level process rewards, which are integrated with outcome rewards for policy optimization\.#### Medical RAG\.

Large language models have demonstrated promising performance on medical question answering benchmarks, as shown by systems such as Med\-PaLM\(Singhal et al\.,[2023a](https://arxiv.org/html/2607.01440#bib.bib24)\), Meditron\(Chen et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib3)\), and Med\-Gemini\(Saab et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib19)\)\. However, these models remain prone to hallucination, motivating the adoption of retrieval\-augmented generation \(RAG\) to ground responses in retrieved medical documents and improve factuality\(Lewis et al\.,[2021](https://arxiv.org/html/2607.01440#bib.bib16); Xiong et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib35); Yang et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib36); Sun et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib28)\)\. Recent work such as MedCite\(Wang et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib31)\)further emphasizes response verifiability by adapting citation generation and retrieval strategies\(Gao et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib7); Huo et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib9); Ye et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib37)\)to the medical domain with specialized evaluation protocols, improving attribution to external medical knowledge sources\. Nevertheless, existing approaches primarily focus on the factuality of final responses, while largely overlooking the faithfulness of the intermediate reasoning process, which may still lead to unreliable clinical conclusions\.

#### Medical Reasoning\.

Clinical decision\-making often requires complex multi\-step reasoning over patient evidence during diagnosis and treatment planning\. Prior work such as MedPrompt\(Nori et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib17)\)leverages Chain\-of\-Thought \(CoT\) prompting\(Wei et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib34)\)to improve medical reasoning capabilities of general\-purpose LLMs during inference\. Similarly, Huatuo\-o1\(Chen et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib2)\)iteratively improves multi\-step reasoning trajectories through verifier\-guided feedback\.

More recently, agentic medical systems have begun integrating reinforcement learning with RAG and process reward modeling using LLM\-as\-a\-judge frameworks to improve reasoning quality\(Jin et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib11); Yun et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib38); Jiang et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib10)\)\. Different from prior work, our method incorporates realistic evidence\-based medicine workflows into a reinforcement learning framework with fine\-grained step\-level rewards\. Guided by clinician\-curated rubrics, our framework explicitly promotes faithful and trustworthy reasoning processes that remain grounded in supporting medical evidence throughout the path toward the final clinical conclusion\.

## 3Preliminaries

#### Reasoning Environment\.

We formulate evidence\-based medical reasoning as an agent–environment interaction\. Given a clinical questionxx, the initial states1s\_\{1\}is conditioned onxx\. At each turnkk, the model observessks\_\{k\}and generatesyk=\(tk,ak\)y\_\{k\}=\(t\_\{k\},a\_\{k\}\), wheretkt\_\{k\}denotes the reasoning trace andak∈\{search,answer\}a\_\{k\}\\in\\\{\\texttt\{search\},\\texttt\{answer\}\\\}is the action\(Jin et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib11)\)\. Asearchaction retrieves external evidenceek=\{ek,j\}je\_\{k\}=\\\{e\_\{k,j\}\\\}\_\{j\}, transitioning to the next statesk\+1s\_\{k\+1\}and yielding rewardrkr\_\{k\}\. The episode terminates when the model selects theansweraction, producing aTT\-turn trajectoryτ=\{\(sk,tk,ak,rk\)\}k=1T\\tau=\\\{\(s\_\{k\},t\_\{k\},a\_\{k\},r\_\{k\}\)\\\}\_\{k=1\}^\{T\}\. To enable citation grounding, each retrieved documentek,je\_\{k,j\}is serialized within an<information\>block and assigned the identifier\[T​kTk\-E​jEj\], which the model is prompted to reference in subsequent reasoning\.

#### Outcome\-Reward RL\.

The outcome reward for a trajectoryτ\\tauis defined asRO​\(τ\)=∑k=1TrkOR^\{O\}\(\\tau\)=\\sum\_\{k=1\}^\{T\}r^\{O\}\_\{k\}\. In the standard GRPO setting, reward is assigned only at the end of the episode\(Shao et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib22)\):rkO=0r^\{O\}\_\{k\}=0fork<Tk<T, whilerTOr^\{O\}\_\{T\}is a binary signal indicating the correctness of the final answer\. For each question, GRPO samplesNNcandidate episodes\{τi\}i=1N\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}from the policyπθ​\(yk∣sk,x\)\\pi\_\{\\theta\}\(y\_\{k\}\\mid s\_\{k\},x\)and computes group\-normalized episode rewards:

AiO=RO​\(τi\)−mean​\(\{RO​\(τj\)\}j=1N\)std​\(\{RO​\(τj\)\}j=1N\)\.A\_\{i\}^\{O\}=\\frac\{R^\{O\}\(\\tau\_\{i\}\)\-\\mathrm\{mean\}\\bigl\(\\\{R^\{O\}\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{N\}\\bigr\)\}\{\\mathrm\{std\}\\bigl\(\\\{R^\{O\}\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{N\}\\bigr\)\}\.\(1\)This yields an episode\-level outcome advantage shared across the entire trajectory\.

## 4FaithMed: Towards Fine\-Grained Process Reward in Medical Reasoning

We present FaithMed, which trains LLMs with rubric\-guided step\-level process rewards for evidence\-based medical reasoning\. We first describe how the evaluation criteria are designed and refined in Section[4\.1](https://arxiv.org/html/2607.01440#S4.SS1)\. We then define the step\-level process reward in Section[4\.2](https://arxiv.org/html/2607.01440#S4.SS2)and show how it is incorporated into policy optimization in Section[4\.3](https://arxiv.org/html/2607.01440#S4.SS3)\.

### 4\.1Rubrics Design and Refinement

We first define the process\-reward dimensions following the principles of evidence\-based medicine\(Sackett et al\.,[1996](https://arxiv.org/html/2607.01440#bib.bib20); Dusin et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib4)\):Ask,Acquire,Appraise,Apply, andAssess\. Specifically, \(1\)Askevaluates whether the information need is formulated as a clear clinical question; \(2\)Acquiremeasures whether the model retrieves relevant external evidence; \(3\)Appraiseevaluates whether the retrieved evidence is critically assessed for relevance and support; \(4\)Applymeasures whether the evidence is integrated with the clinical context; and \(5\)Assessevaluates whether the overall reasoning process and final decision are reviewed\. For each dimension, we first design seed criteria and then decompose them into binary checklist\-style rubrics following CheckEval\(Lee et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib15)\)\.

To reduce redundancy and eliminate uninformative criteria, we use teacher\-generated medical QA reasoning trajectories to iteratively refine the curated rubric set until convergence\. Specifically, we adopt a metric\-driven recursive rubric\-refinement procedure\. In each iteration, an LLM judge scores all trajectory–rubric pairs, after which we compute rubric\-level statistics, including non\-applicability rate,Metrate, and pairwise Pearson correlation between rubrics\.

These rubric\-level statistics are then used to determine the refinement action: \(1\)Mergerubrics with high pairwise correlation, as they capture overlapping reasoning behaviors; \(2\)Removeweakly discriminative rubrics with nearly constant labels, including rubrics that are almost alwaysMetorUnmet; and \(3\)Rewriterubrics with high non\-applicability rates, which typically reflect overly narrow or ambiguous preconditions\. The refinement process terminates when no active rubric satisfies any refinement criterion\.MergeandRewriteoperations are automatically performed by an LLM editor\. Finally, a medical expert reviews the refined rubric set and performs final edits, yielding the rubrics used for process evaluation; the final rubric set is provided in Appendix[A\.1](https://arxiv.org/html/2607.01440#A1.SS1)\.

### 4\.2Step\-level Process Reward

Outcome rewards only indicate whether the final answer is correct, but provide limited guidance on whether intermediate reasoning decisions are faithful to the available evidence\. We therefore construct a process reward for each generated stepyk=\(tk,ak\)y\_\{k\}=\(t\_\{k\},a\_\{k\}\)by judging the step against the rubric dimensions that are relevant to its local context\. Lete0=∅e\_\{0\}=\\emptyset\. For a stepkk, we assign a dimension setDkD\_\{k\}as follows: the initial step includesAsk; asearchaction includesAcquire; if the model has observed non\-empty retrieved informationek−1e\_\{k\-1\}, the step includesAppraiseandApply; and anansweraction includesAssess\. These assignments are additive, so a step can be evaluated by multiple dimensions\.

Then, we construct the applicable rubric set and evaluate the corresponding reasoning step:

ℛk\\displaystyle\\mathcal\{R\}\_\{k\}=⋃d∈Dkℛd,\\displaystyle=\\bigcup\_\{d\\in D\_\{k\}\}\\mathcal\{R\}\_\{d\},\(2\)Jk\\displaystyle J\_\{k\}=J​\(yk,ek−1,ℛk\)∈\{0,1\}\|ℛk\|,\\displaystyle=J\(y\_\{k\},e\_\{k\-1\},\\mathcal\{R\}\_\{k\}\)\\in\\\{0,1\\\}^\{\|\\mathcal\{R\}\_\{k\}\|\},\(3\)whereℛd\\mathcal\{R\}\_\{d\}denotes the refined rubrics for dimensiondd, andJkJ\_\{k\}denotes the binary rubric scores returned by the judge for stepkk\. For trajectoryii, we aggregate the applicable rubric scores into a scalar step\-level process rewardri,kP=Agg​\(Ji,k\)r^\{P\}\_\{i,k\}=\\mathrm\{Agg\}\(J\_\{i,k\}\), whereAgg​\(⋅\)\\mathrm\{Agg\}\(\\cdot\)denotes mean\-centered rubric aggregation\.

To obtain comparable process rewards, we group reasoning steps with similar observation contexts and retrieved evidence as anchor\-state, following the strategy of GiGPO\(Feng et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib5)\)\. The process advantage is computed using group\-normalized step\-level rewards:

Ai,kP=ri,kP−mean​\(𝒢​\(s~i,k\)\)std​\(𝒢​\(s~i,k\)\)\+ϵ,A^\{P\}\_\{i,k\}=\\frac\{r^\{P\}\_\{i,k\}\-\\mathrm\{mean\}\\\!\\left\(\\mathcal\{G\}\(\\tilde\{s\}\_\{i,k\}\)\\right\)\}\{\\mathrm\{std\}\\\!\\left\(\\mathcal\{G\}\(\\tilde\{s\}\_\{i,k\}\)\\right\)\+\\epsilon\},\(4\)where𝒢​\(s~i,k\)\\mathcal\{G\}\(\\tilde\{s\}\_\{i,k\}\)denotes the multiset of process rewards in the anchor group associated with anchor states~i,k\\tilde\{s\}\_\{i,k\}\. This yields fine\-grained comparative process feedback, where a search or answer step is rewarded when it better satisfies the relevant rubrics than alternative steps under a similar observation context\.

### 4\.3FaithMed Policy Optimization

Outcome\-only RL assigns the same episode\-level signal to every step in a trajectory\. In FaithMed, the final\-answer rewardrOr^\{O\}and rubric\-guided process rewardrPr^\{P\}are treated as separate supervision signals:rOr^\{O\}defines the episode\-level outcome advantageAiOA\_\{i\}^\{O\}in Eq\.[1](https://arxiv.org/html/2607.01440#S3.E1), whilerPr^\{P\}defines the step\-level process advantageAi,kPA^\{P\}\_\{i,k\}in Eq\.[4](https://arxiv.org/html/2607.01440#S4.E4)\. The advantage for stepkkin trajectoryiiis computed as

Ai,k=AiO\+λ​Ai,kP,A\_\{i,k\}=A^\{O\}\_\{i\}\+\\lambda A^\{P\}\_\{i,k\},\(5\)whereλ\\lambdacontrols the contribution of rubric\-guided process feedback\. The first term encourages trajectories that reach the correct final answer, while the second term provides additional credit to intermediate reasoning steps that better satisfy the process rubrics under a similar evidence context\. We then optimizeAi,kA\_\{i,k\}using the standard GRPO clipped objective over the generated tokens in stepyi,ky\_\{i,k\}, together with the standard reference\-policy KL penalty\. This enables FaithMed to jointly optimize final\-answer correctness and intermediate evidence\-based medicine behaviors\.

Table 1:Main benchmark results\. FaithMed here uses the no\-SFT recipe\. GRPO uses outcome rewards only\.§denotes reference LLM values reported by Medmarks\-Verifiable\(Warner et al\.,[2026](https://arxiv.org/html/2607.01440#bib.bib33)\)\.†denotes in\-domain datasets and∗denotes out\-of\-domain datasets\.Table 2:Process scores across five dimensions\. \+ Ep\. PR uses episode\-level process reward, while FaithMed uses step\-level process reward\.Table 3:Training\-recipe ablation\. OR denotes outcome reward, PR denotes process reward, and Ep\. and Step indicate episode\-level and step\-level PR, respectively\.

## 5Experiments Setup

#### Datasets\.

We evaluate on a diverse suite of medical benchmarks that cover complementary types of medical tasks: \(1\) knowledge\-intensive question answering, including HeadQA\(Vilares and Gómez\-Rodríguez,[2019](https://arxiv.org/html/2607.01440#bib.bib30)\), MedMCQA\(Pal et al\.,[2022](https://arxiv.org/html/2607.01440#bib.bib18)\), and MMLU\-Pro\-Health\(Wang et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib32)\); \(2\) expert\-level medical reasoning, represented by MedXpertQA\(Zuo et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib39)\); \(3\) quantitative medical calculation, represented by MedCalc\-Bench\(Khandekar et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib14)\); and \(4\) clinical application question answering, including MedQA\(Jin et al\.,[2021](https://arxiv.org/html/2607.01440#bib.bib12)\)and MedBullets5op\(Chen et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib1)\)\. We use HeadQA, MedMCQA, MedCalc\-Bench, and MedQA for training and in\-distribution evaluation, and reserve MMLU\-Pro\-Health, MedXpertQA, and MedBullets5opfor out\-of\-distribution evaluation\. To reduce near\-duplicate leakage, we apply 16\-gram decontamination against the held\-out evaluation sets\.

#### Baselines\.

We compare FaithMed against three categories of baselines: \(1\) reference LLMs, including general\-purpose, and medical models; \(2\) evidence\-utilization strategies, including MedRAG\-style pre\-retrieval\(Xiong et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib35)\)and agentic search prompting; and \(3\) training variants, including GRPO with episode\-level outcome rewards and alternative process\-reward formulations\.

#### Rubrics Refinement\.

We use 300 medical QA reasoning trajectories generated by the teacher model to refine the rubric set\. The refinement criteria are instantiated as follows:Mergeis applied when any pair of rubrics has an absolute Pearson correlation above 0\.70,Rewritewhen the non\-applicability rate exceeds 0\.80, andRemovewhen theMetrate is above 0\.95 or below 0\.05\. Starting from 26 seed rubrics, the refinement procedure converges after three iterations, yielding 15 rubrics\.

#### Training Recipes\.

\(1\)SFT training\.We construct supervised trajectories using DeepSeek\-R1 with agentic search prompting on 2,650 questions from MedQA, MedMCQA, HeadQA, and MedCalc\-Bench\. For each question, we retain 6 trajectories with correct answers and valid formats; questions with fewer than 6 valid trajectories within 20 attempts are filtered out\. We fine\-tune Qwen3 backbones for one epoch using a learning rate of5×10−55\\times 10^\{\-5\}, cosine decay, and a 1% linear warmup ratio\. \(2\)RL training\.We perform RL training for 300 steps with a batch size of 32 and 8 rollouts per prompt\. We use Gemini\-2\.5\-Flash\-Lite as the process\-reward judge, setλ=0\.05\\lambda=0\.05for step\-level process rewards, and use a maximum response length of 3072\. For anchor\-state grouping, we use a fuzzy string similarity threshold of 0\.8\. Additional implementation details are provided in the appendix[A\.2](https://arxiv.org/html/2607.01440#A1.SS2)\.

## 6Results

In this section, we provide a comprehensive evaluation of FaithMed\. We first report benchmark performance and evidence\-based medicine reasoning scores in Section[6\.1](https://arxiv.org/html/2607.01440#S6.SS1)\. We then present ablation studies over training recipes and rubric dimensions in Section[6\.2](https://arxiv.org/html/2607.01440#S6.SS2), followed by diagnostics of process\-reward granularity and advantage grouping in Section[6\.3](https://arxiv.org/html/2607.01440#S6.SS3)\. Finally, we present case studies that illustrate evidence\-based behaviors in Section[6\.4](https://arxiv.org/html/2607.01440#S6.SS4)\.

### 6\.1Overall Results

Table[1](https://arxiv.org/html/2607.01440#S4.T1)shows that FaithMed consistently improves benchmark answer accuracy across both backbones\. The baseline comparisons indicate that external evidence is useful, but the best access strategy depends on model capacity: MedRAG improves the 1\.7B model more than agentic search \(40\.3% vs\. 38\.6%\), whereas agentic search is more effective for the 4B model \(52\.5% vs\. 50\.1%\), suggesting that smaller models may struggle to execute agentic workflows while benefiting more from retrieved medical knowledge\.

Beyond evidence access, outcome\-only RL improves performance, but FaithMed yields larger gains, raising benchmark answer accuracy from 41\.0% to 47\.8% on 1\.7B and from 56\.5% to 61\.2% on 4B\. This shows that learning a faithful evidence\-based reasoning process improves task success, not only reasoning quality; notably, FaithMed\-4B surpasses the larger Qwen3 8B reference model \(61\.2% vs\. 60\.7%\)\.

Beyond answer accuracy, Table[2](https://arxiv.org/html/2607.01440#S4.T2)shows that FaithMed substantially improves the evidence\-based medicine reasoning process itself\. Improvements are strongest on dimensions requiring active evidence use: for both backbones, FaithMed improves Acquire by roughly 23–24%, Appraise by roughly 21–22%, and Apply by roughly 18% over the base model\. We also compare against an episode\-level process\-reward baseline, which provides episode\-level process feedback without assigning it to specific reasoning steps\. FaithMed consistently outperforms this baseline on both backbones, demonstrating the benefit of fine\-grained faithful reasoning supervision\. FaithMed\-4B even matches the teacher model DeepSeek\-R1 on 3 out of 5 dimensions\.

### 6\.2Ablation Study

Table 4:Process reward dimension ablation results\. Ablation rows report changes from the full model\.![Refer to caption](https://arxiv.org/html/2607.01440v1/x3.png)Figure 3:Rubric\-score curves across seven medical benchmarks\. Scores are averaged over the five rubric dimensions for Qwen3\-1\.7B trained with SFT \+ RL using episode\-level or step\-level process reward\.Table 5:Case studies of process\-dimension behaviors\. High\-quality responses are selected from FaithMed\-4B, while lower\-quality responses are selected from Qwen3\-4B with agentic search\. Green and red highlights mark phrases most aligned or misaligned with the corresponding rubric dimension\. Ellipses mark omissions made for space\.![Refer to caption](https://arxiv.org/html/2607.01440v1/x4.png)\(a\)Episode\-level signal
![Refer to caption](https://arxiv.org/html/2607.01440v1/x5.png)\(b\)RA and AG ablation

Figure 4:Diagnostics for reward assignment \(RA\) and advantage grouping \(AG\)\.#### Step\-level process reward is the most reliable training recipe\.

We first examine which training recipe is responsible for the gains of FaithMed\. Table[3](https://arxiv.org/html/2607.01440#S4.T3)varies \(1\) supervised fine\-tuning and RL rewards: \(2\) outcome reward and \(3\) episode\- versus step\-level process reward\. Outcome\-only RL improves over the untrained policy, but the largest gains come from adding step\-level process reward\. Without SFT, step\-level process reward improves overall accuracy by 9\.2 points for the 1\.7B backbone and 8\.7 points for the 4B backbone\. The same pattern holds after SFT, where it adds 6\.1 and 5\.3 points over the SFT\-only 1\.7B and 4B models, respectively\.

Combining outcome reward with episode\-level process reward is less reliable: it improves some no\-SFT settings but offers little or negative gain after SFT\. This pattern supports a central design choice of FaithMed: process supervision should be assigned at finer granularity, so each reasoning step can be compared with similar step groups on the corresponding evidence\-based behavior\.

#### Rubric dimensions target distinct reasoning stages\.

Table[4](https://arxiv.org/html/2607.01440#S6.T4)then ablates the reward dimensions themselves\. The largest drops occur on the ablated dimension: removing Acquire, Appraise, Apply, and Assess reduces the corresponding score by 8\.28, 9\.35, 8\.51, and 8\.44 points, respectively\. This concentration of errors shows that the reward dimensions target distinct parts of the reasoning trace\. At the same time, the effects are not fully isolated\. Removing Ask or Acquire also lowers later stages such as Appraise and Assess, suggesting that evidence\-based medical reasoning is sequential and interdependent: poor question formulation or evidence gathering can limit the quality of later evidence evaluation and clinical application\.

### 6\.3Diagnosing Step\-Level Process Reward

The ablations above show that step\-level process reward is more reliable than episode\-level process reward\. Figure[3](https://arxiv.org/html/2607.01440#S6.F3)confirms this trend across benchmarks: it provides more consistent reasoning\-quality gains than the episode\-level baseline\. To understand why step\-level process reward works better, we diagnose this advantage through two components of process\-reward training: reward assignment \(RA\), which assigns rubric feedback to steps or episodes, and advantage grouping \(AG\), which normalizes advantages within anchor\-state step groups or rollout groups\.

#### Episode\-level assignment creates off\-step credit\.

Episode\-level reward assignment can give a step feedback from rubrics that the step does not directly control\. Figure[4\(a\)](https://arxiv.org/html/2607.01440#S6.F4.sf1)shows this credit\-assignment mismatch by decomposing the rubric signal assigned to each step into on\-step and off\-step components\. This mismatch is especially problematic for early search actions, which can be covered by only a small number of corresponding rubrics\. Their updates can therefore be dominated by noisy off\-step feedback from later evidence appraisal or answer quality, causing the model to learn unstable signals at the start of the reasoning process\.

#### Grouping helps only with step\-aligned rewards\.

To separate the role of grouping from reward assignment, we compare an intermediate setting that uses step\-level advantage grouping but keeps episode\-level reward assignment\. In Figure[4\(b\)](https://arxiv.org/html/2607.01440#S6.F4.sf2), this setting provides only limited improvement, whereas combining step\-level reward assignment with step\-level advantage grouping substantially improves performance\. This suggests that the main gain comes from aligning both the reward signal and the comparison group with the reasoning step being optimized\.

### 6\.4Case Study

#### Case studies reveal concrete reasoning differences\.

Beyond aggregate rubric scores, we examine representative trajectories to identify what the process improvements look like in model outputs\. Table[5](https://arxiv.org/html/2607.01440#S6.T5)shows that the FaithMed responses explicitly identify decisive clinical clues, formulate decision\-directed searches, prioritize directly supporting evidence, and check whether retrieved guidance applies to the patient\. In contrast, the agentic\-search baseline often uses plausible but underspecified heuristics, generic search queries, or evidence that is not logically aligned with the clinical decision\. These examples show how the rubric gains in Table[2](https://arxiv.org/html/2607.01440#S4.T2)correspond to concrete improvements in the reasoning trace rather than only higher final\-answer accuracy\.

## 7Conclusion

We introduced FaithMed, a rubric\-guided training framework for faithful evidence\-based medical reasoning\. Instead of rewarding only final answers, FaithMed decomposes reasoning into evidence\-based medicine dimensions and uses step\-level process rewards to give credit to local actions that ask clinically precise questions, acquire relevant evidence, appraise retrieved passages, apply evidence to the patient context, and assess the final decision\. Across seven medical benchmarks, FaithMed improves both 1\.7B and 4B Qwen3 backbones and outperforms retrieval\-only and agentic\-search baselines\. Our ablations show that step\-level process rewards provide more consistent gains than episode\-level process rewards, and rubric\-level analyses confirm that these gains reflect improved evidence\-use behaviors rather than only better answer selection\. These results suggest that faithful medical reasoning requires not only access to external knowledge, but also explicit training of how evidence should be sought, judged, and applied\.

## 8Limitations

Our study focuses on textual medical question answering benchmarks, which simplify the complexity of real clinical workflows involving longitudinal records, laboratory results, imaging, physical examinations, and institution\-specific constraints\. In addition, our process supervision depends on clinician\-reviewed rubrics and LLM\-based judges, which are useful but still imperfect proxies for faithful medical reasoning; models may learn to satisfy rubric criteria without fully eliminating unsupported claims\. The current experiments are also limited to Qwen3 backbones, English medical QA settings, and a fixed retrieval pool, so broader evaluation across model families, languages, specialties, and evolving medical evidence sources remains future work\.

## 9Ethical Considerations

This work is intended as research on improving the faithfulness of medical reasoning in language models and should not be used as a substitute for professional diagnosis, treatment planning, or patient\-specific medical advice\. We evaluate on existing medical QA benchmarks and use publicly available medical corpora for retrieval; we do not collect new patient records or personal health information\. However, models trained or evaluated in this setting can still produce incorrect answers, misleading explanations, or unsupported medical claims, and any real\-world use would require rigorous clinical validation, privacy safeguards for user\-provided health information, and oversight by qualified medical professionals\. We also note that medical benchmarks and retrieval corpora may encode demographic, geographic, linguistic, or institutional biases, so improved benchmark performance does not guarantee equitable behavior across patient populations\.

## References

- Chen et al\. \(2025\)Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze\. 2025\.[Benchmarking large language models on answering and explaining challenging medical questions](https://doi.org/10.18653/v1/2025.naacl-long.182)\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 3563–3599, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Chen et al\. \(2024\)Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang\. 2024\.[Huatuogpt\-o1, towards medical complex reasoning with llms](https://arxiv.org/abs/2412.18925)\.*Preprint*, arXiv:2412\.18925\.
- Chen et al\. \(2023\)Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary\-Anne Hartley, Martin Jaggi, and Antoine Bosselut\. 2023\.[Meditron\-70b: Scaling medical pretraining for large language models](https://arxiv.org/abs/2311.16079)\.*Preprint*, arXiv:2311\.16079\.
- Dusin et al\. \(2023\)Jarrod Dusin, Andrea Melanson, and Lisa Mische\-Lawson\. 2023\.[Evidence\-based practice models and frameworks in the healthcare setting: a scoping review](https://doi.org/10.1136/bmjopen-2022-071188)\.*BMJ Open*, 13\(5\):e071188\.
- Feng et al\. \(2025\)Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An\. 2025\.[Group\-in\-group policy optimization for LLM agent training](https://arxiv.org/abs/2505.10978)\.*arXiv preprint arXiv:2505\.10978*\.
- Gao et al\. \(2022\)Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan\. 2022\.[Tevatron: An efficient and flexible toolkit for dense retrieval](https://arxiv.org/abs/2203.05765)\.*arXiv preprint arXiv:2203\.05765*\.
- Gao et al\. \(2023\)Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen\. 2023\.[Enabling large language models to generate text with citations](https://doi.org/10.18653/v1/2023.emnlp-main.398)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 6465–6488, Singapore\. Association for Computational Linguistics\.
- Gruppen \(2017\)Larry D Gruppen\. 2017\.[Clinical reasoning: Defining it, teaching it, assessing it, studying it](https://doi.org/10.5811/westjem.2016.11.33191)\.*The Western Journal of Emergency Medicine*, 18\(1\):4–7\.
- Huo et al\. \(2023\)Siqing Huo, Negar Arabzadeh, and Charles Clarke\. 2023\.[Retrieving supporting evidence for generative question answering](https://doi.org/10.1145/3624918.3625336)\.In*Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region*, SIGIR\-AP ’23, page 11–20\. ACM\.
- Jiang et al\. \(2025\)Shuyang Jiang, Yusheng Liao, Zhe Chen, Ya Zhang, Yanfeng Wang, and Yu Wang\. 2025\.[Meds3: Towards medical slow thinking with self\-evolved soft dual\-sided process supervision](https://arxiv.org/abs/2501.12051)\.*Preprint*, arXiv:2501\.12051\.
- Jin et al\. \(2025\)Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han\. 2025\.[Search\-R1: Training LLMs to reason and leverage search engines with reinforcement learning](https://arxiv.org/abs/2503.09516)\.*arXiv preprint arXiv:2503\.09516*\.
- Jin et al\. \(2021\)Di Jin, Eileen Pan, Nassim Oufattole, Wei\-Hung Weng, Hanyi Fang, and Peter Szolovits\. 2021\.[What disease does this patient have? a large\-scale open domain question answering dataset from medical exams](https://doi.org/10.3390/app11146421)\.*Applied Sciences*, 11\(14\):6421\.
- Jin et al\. \(2019\)Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu\. 2019\.[PubMedQA: A dataset for biomedical research question answering](https://doi.org/10.18653/v1/D19-1259)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 2567–2577, Hong Kong, China\. Association for Computational Linguistics\.
- Khandekar et al\. \(2024\)Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina S\. Applebaum, Zain Anwar, Maame Sarfo\-Gyamfi, Conrad W\. Safranek, Abid A\. Anwar, Andrew Zhang, Aidan Gilson, Maxwell B\. Singer, Amisha Dave, Andrew Taylor, Aidong Zhang, Qingyu Chen, and Zhiyong Lu\. 2024\.[Medcalc\-bench: Evaluating large language models for medical calculations](https://papers.nips.cc/paper_files/paper/2024/hash/99e81750f3fdfcaf9613db2dbf4bd623-Abstract-Datasets_and_Benchmarks_Track.html)\.In*Advances in Neural Information Processing Systems 37*\.
- Lee et al\. \(2025\)Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim\. 2025\.[CheckEval: A reliable LLM\-as\-a\-judge framework for evaluating text generation using checklists](https://doi.org/10.18653/v1/2025.emnlp-main.796)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 15771–15798, Suzhou, China\. Association for Computational Linguistics\.
- Lewis et al\. \(2021\)Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela\. 2021\.[Retrieval\-augmented generation for knowledge\-intensive nlp tasks](https://arxiv.org/abs/2005.11401)\.*Preprint*, arXiv:2005\.11401\.
- Nori et al\. \(2023\)Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz\. 2023\.[Can generalist foundation models outcompete special\-purpose tuning? case study in medicine](https://arxiv.org/abs/2311.16452)\.*Preprint*, arXiv:2311\.16452\.
- Pal et al\. \(2022\)Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu\. 2022\.[Medmcqa: A large\-scale multi\-subject multi\-choice dataset for medical domain question answering](https://proceedings.mlr.press/v174/pal22a.html)\.In*Proceedings of the Conference on Health, Inference, and Learning*, volume 174 of*Proceedings of Machine Learning Research*, pages 248–260\. PMLR\.
- Saab et al\. \(2024\)Khaled Saab, Tao Tu, Wei\-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu\-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G\. T\. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, Daniel McDuff, Le Hou, Tomer Golany, Luyang Liu, Jean baptiste Alayrac, Neil Houlsby, Nenad Tomasev, Jan Freyberg, Charles Lau, Jonas Kemp, Jeremy Lai, Shekoofeh Azizi, Kimberly Kanada, SiWai Man, Kavita Kulkarni, Ruoxi Sun, Siamak Shakeri, Luheng He, Ben Caine, Albert Webson, Natasha Latysheva, Melvin Johnson, Philip Mansfield, Jian Lu, Ehud Rivlin, Jesper Anderson, Bradley Green, Renee Wong, Jonathan Krause, Jonathon Shlens, Ewa Dominowska, S\. M\. Ali Eslami, Katherine Chou, Claire Cui, Oriol Vinyals, Koray Kavukcuoglu, James Manyika, Jeff Dean, Demis Hassabis, Yossi Matias, Dale Webster, Joelle Barral, Greg Corrado, Christopher Semturs, S\. Sara Mahdavi, Juraj Gottweis, Alan Karthikesalingam, and Vivek Natarajan\. 2024\.[Capabilities of gemini models in medicine](https://arxiv.org/abs/2404.18416)\.*Preprint*, arXiv:2404\.18416\.
- Sackett et al\. \(1996\)David L Sackett, William MC Rosenberg, J A Muir Gray, R Brian Haynes, and W Scott Richardson\. 1996\.[Evidence based medicine: what it is and what it isn’t](https://doi.org/10.1136/bmj.312.7023.71)\.*BMJ*, 312\(7023\):71–72\.
- Schwartz and Elstein \(2008\)Alan Schwartz and Arthur S\. Elstein\. 2008\.Clinical reasoning in medicine\.In Joy Higgs, Mark A\. Jones, Stephen Loftus, and Nicole Christensen, editors,*Clinical Reasoning in the Health Professions*, 3 edition\. Elsevier, Edinburgh\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\. 2024\.[DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)\.*arXiv preprint arXiv:2402\.03300*\.
- Sheng et al\. \(2025\)Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu\. 2025\.[Hybridflow: A flexible and efficient RLHF framework](https://doi.org/10.1145/3689031.3696075)\.In*Proceedings of the Twentieth European Conference on Computer Systems*, EuroSys ’25, Rotterdam, Netherlands\. Association for Computing Machinery\.
- Singhal et al\. \(2023a\)Karan Singhal, Shekoofeh Azizi, Tao Tu, S\. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole\-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner\-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S\. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomašev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan\. 2023a\.[Large language models encode clinical knowledge](https://doi.org/10.1038/s41586-023-06291-2)\.*Nature*, 620\(7972\):172–180\.
- Singhal et al\. \(2023b\)Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole\-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S\. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S\. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan\. 2023b\.[Towards expert\-level medical question answering with large language models](https://arxiv.org/abs/2305.09617)\.*Preprint*, arXiv:2305\.09617\.
- Subramanya et al\. \(2019\)Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi\. 2019\.[DiskANN: Fast accurate billion\-point nearest neighbor search on a single node](https://papers.nips.cc/paper/9527-diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node)\.In*Advances in Neural Information Processing Systems 32*\.
- Sun et al\. \(2024\)Liwen Sun, Abhineet Agarwal, Aaron Kornblith, Bin Yu, and Chenyan Xiong\. 2024\.[Ed\-copilot: Reduce emergency department wait time with language model diagnostic assistance](https://arxiv.org/abs/2402.13448)\.*Preprint*, arXiv:2402\.13448\.
- Sun et al\. \(2025\)Liwen Sun, James Zhao, Megan Han, and Chenyan Xiong\. 2025\.[Fact\-aware multimodal retrieval augmentation for accurate medical radiology report generation](https://arxiv.org/abs/2407.15268)\.*Preprint*, arXiv:2407\.15268\.
- Temsah et al\. \(2023\)Mohamad\-Hani Temsah, Fadi Aljamaan, Khalid H\. Malki, Khalid Alhasan, Ibraheem Altamimi, Razan Aljarbou, Faisal Bazuhair, Abdulmajeed Alsubaihin, Naif Abdulmajeed, Fatimah S\. Alshahrani, Reem Temsah, Turki Alshahrani, Lama Al\-Eyadhy, Serin Mohammed Alkhateeb, Basema Saddik, Rabih Halwani, Amr Jamal, Jaffar A\. Al\-Tawfiq, and Ayman Al\-Eyadhy\. 2023\.[Chatgpt and the future of digital health: A study on healthcare workers’ perceptions and expectations](https://doi.org/10.3390/healthcare11131812)\.*Healthcare*, 11\(13\):1812\.
- Vilares and Gómez\-Rodríguez \(2019\)David Vilares and Carlos Gómez\-Rodríguez\. 2019\.[HEAD\-QA: A healthcare dataset for complex reasoning](https://doi.org/10.18653/v1/P19-1092)\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 960–966, Florence, Italy\. Association for Computational Linguistics\.
- Wang et al\. \(2025\)Xiao Wang, Mengjue Tan, Qiao Jin, Guangzhi Xiong, Yu Hu, Aidong Zhang, Zhiyong Lu, and Minjia Zhang\. 2025\.[MedCite: Can language models generate verifiable text for medicine?](https://doi.org/10.18653/v1/2025.findings-acl.967)In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 18891–18913, Vienna, Austria\. Association for Computational Linguistics\.
- Wang et al\. \(2024\)Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\. 2024\.[MMLU\-pro: A more robust and challenging multi\-task language understanding benchmark](https://papers.nips.cc/paper_files/paper/2024/hash/ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html)\.In*Advances in Neural Information Processing Systems 37*\.
- Warner et al\. \(2026\)Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer, Aymane Ouraq, Saurav Panigrahi, Geetu Ambwani, Kunal Bagga, Nikhil Khandekar, Arya Hariharan, Nishant Mishra, Manish Ram, Shamus Sim Zi Yang, Ahmed Essouaied, Adepoju Jeremiah Moyondafoluwa, Robert Scholz, Bofeng Huang, Molly Beavers, Srishti Gureja, Anish Mahishi, Sameed Khan, Maxime Griot, Hunar Batra, Jean\-Benoit Delbrouck, Siddhant Bharadwaj, Ronald Clark, Ashish Vashist, Anas Zafar, Leema Krishna Murali, Harsh Deshpande, Ameen Patel, William Brown, Johannes Hagemann, Connor Lane, Paul Steven Scotti, and Tanishq Mathew Abraham\. 2026\.[Medmarks: A comprehensive open\-source llm benchmark suite for medical tasks](https://arxiv.org/abs/2605.01417)\.*Preprint*, arXiv:2605\.01417\.
- Wei et al\. \(2023\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou\. 2023\.[Chain\-of\-thought prompting elicits reasoning in large language models](https://arxiv.org/abs/2201.11903)\.*Preprint*, arXiv:2201\.11903\.
- Xiong et al\. \(2024\)Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang\. 2024\.[Benchmarking retrieval\-augmented generation for medicine](https://doi.org/10.18653/v1/2024.findings-acl.372)\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 6233–6251, Bangkok, Thailand\. Association for Computational Linguistics\.
- Yang et al\. \(2024\)Hua Yang, Shilong Li, and Teresa Gonçalves\. 2024\.[Enhancing biomedical question answering with large language models](https://doi.org/10.3390/info15080494)\.*Information*, 15\(8\)\.
- Ye et al\. \(2024\)Xi Ye, Ruoxi Sun, Sercan Ö\. Arik, and Tomas Pfister\. 2024\.[Effective large language model adaptation for improved grounding and citation generation](https://arxiv.org/abs/2311.09533)\.*Preprint*, arXiv:2311\.09533\.
- Yun et al\. \(2025\)Jaehoon Yun, Jiwoong Sohn, Jungwoo Park, Hyunjae Kim, Xiangru Tang, Yanjun Shao, Yonghoe Koo, Minhyeok Ko, Qingyu Chen, Mark Gerstein, Michael Moor, and Jaewoo Kang\. 2025\.[Med\-prm: Medical reasoning models with stepwise, guideline\-verified process rewards](https://arxiv.org/abs/2506.11474)\.*Preprint*, arXiv:2506\.11474\.
- Zuo et al\. \(2025\)Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou\. 2025\.[Medxpertqa: Benchmarking expert\-level medical reasoning and understanding](https://arxiv.org/abs/2501.18362)\.*arXiv preprint arXiv:2501\.18362*\.

## Appendix AAppendix

### A\.1Rubric Details

We list our evidence\-based medicine rubrics in Table[6](https://arxiv.org/html/2607.01440#A1.T6),[7](https://arxiv.org/html/2607.01440#A1.T7)\. Each rubric is binary\-scored as MET or UNMET with examples from teacher\-generated medical QA reasoning trajectories\.

Table 6:Ask and Acquire rubrics used to evaluate evidence\-based medical reasoning\.Table 7:Appraise, Apply, and Assess rubrics used to evaluate evidence\-based medical reasoning\. Each rubric is binary\-scored as MET or UNMET with examples from teacher\-generated medical QA reasoning trajectories\.
### A\.2Implementation Details

Retriever Implementation\.We build the retrieval pool from five medical corpora\. Four corpora—PubMed, StatPearls, medical textbooks, and Wikipedia—are taken from the MedRAG benchmark\(Xiong et al\.,[2024](https://arxiv.org/html/2607.01440#bib.bib35)\)\. We additionally include the clinical guideline corpus released with MediTron\-70B\(Chen et al\.,[2023](https://arxiv.org/html/2607.01440#bib.bib3)\)\. We segment all source documents into short snippets and encode them with MiniCPM\-Embedding\-Light using the Tevatron dense\-retrieval pipeline\(Gao et al\.,[2022](https://arxiv.org/html/2607.01440#bib.bib6)\)\. The resulting snippet embeddings are sharded and served with DiskANN indices for efficient vector search\(Subramanya et al\.,[2019](https://arxiv.org/html/2607.01440#bib.bib26)\)\.

RL Implementation\.We denote the score for rubric

m∈ℛkm\\in\\mathcal\{R\}\_\{k\}as

Jm​\(yk,ek−1,ℛk\)J\_\{m\}\(y\_\{k\},e\_\{k\-1\},\\mathcal\{R\}\_\{k\}\)\. In our retrieval setting, the anchor state

s~i,k\\tilde\{s\}\_\{i,k\}for step

kkin trajectory

iiis the observation\-induced state used for grouping:

s~i,1=x\\tilde\{s\}\_\{i,1\}=xfor the initial step, and

s~i,k=ei,k−1\\tilde\{s\}\_\{i,k\}=e\_\{i,k\-1\}for later steps\. For the

NNtrajectories sampled for the same question

xx, the step\-level group anchored at

s~\\tilde\{s\}is

𝒢​\(s~\)=\{\(i,k\):sim​\(s~i,k,s~\)≥η\},\\mathcal\{G\}\(\\tilde\{s\}\)=\\\{\(i,k\):\\mathrm\{sim\}\(\\tilde\{s\}\_\{i,k\},\\tilde\{s\}\)\\geq\\eta\\\},\(6\)wheresim​\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\)measures anchor\-state similarity andη\\etais the grouping threshold\. Thus, all initial steps for the same question form one group, and later steps are compared with other steps that condition on similar retrieved evidence\.

Before summing rubric scores, we center each rubric within the local group to remove rubric\-specific baselines\. Letμi,k,m\\mu\_\{i,k,m\}denote the mean score of rubricmmamong group members for whichmmis applicable\. The step\-level process reward is then the sum of centered rubric scores:

ri,kP=∑m∈ℛi,k\(Jm​\(yi,k,ei,k−1,ℛi,k\)−μi,k,m\)\.r^\{P\}\_\{i,k\}=\\sum\_\{m\\in\\mathcal\{R\}\_\{i,k\}\}\\left\(J\_\{m\}\(y\_\{i,k\},e\_\{i,k\-1\},\\mathcal\{R\}\_\{i,k\}\)\-\\mu\_\{i,k,m\}\\right\)\.\(7\)This rubric\-level centering reduces sensitivity to rubric\-specific positive rates before scores are accumulated into a step reward\. We then compute the process advantage with a group\-wise scale factor:

Ai,kP=ri,kP−mean​\(\{rj,lP:\(j,l\)∈𝒢​\(s~i,k\)\}\)std​\(\{rj,lP:\(j,l\)∈𝒢​\(s~i,k\)\}\)\+ϵ\.A^\{P\}\_\{i,k\}=\\frac\{r^\{P\}\_\{i,k\}\-\\mathrm\{mean\}\(\\\{r^\{P\}\_\{j,l\}:\(j,l\)\\in\\mathcal\{G\}\(\\tilde\{s\}\_\{i,k\}\)\\\}\)\}\{\\mathrm\{std\}\(\\\{r^\{P\}\_\{j,l\}:\(j,l\)\\in\\mathcal\{G\}\(\\tilde\{s\}\_\{i,k\}\)\\\}\)\+\\epsilon\}\.\(8\)This makes the process feedback comparative: a search, or answer step is rewarded when it better satisfies the relevant rubrics than alternative steps taken under a similar observation context\.

Supervise Finetuning\.We construct supervised trajectories using DeepSeek\-R1 from 1k MedQA, 0\.5k HeadQA, 1k MedMCQA, and 0\.5k MedCalc\-Bench training questions\. We apply rejection sampling by retaining only trajectories with correct answers and valid output formats\. For each question, we collect up to 6 valid trajectories; questions with fewer than 6 valid trajectories after 20 attempts are filtered as hard questions\. In total, the SFT data generation process yields 2,650 questions, 15,900 trajectories, and 28,368 SFT training samples, where each reasoning step is treated as one training sample\. We additionally collect 7,538 rejected trajectories and trajectories from hard questions\.

Reinforcement Learning\.We implement RL training using verl\-agent, an agent\-training extension of verl\(Sheng et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib23); Feng et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib5)\)\. Training is performed for 300 steps with a batch size of 32 and 8 rollouts per prompt\. For advantage grouping, we follow GiGPO\(Feng et al\.,[2025](https://arxiv.org/html/2607.01440#bib.bib5)\)\. For process\-reward scoring, we use Gemini\-2\.5\-Flash\-Lite as the LLM judge\. Rubric evaluation is performed with a single batched judge call per reasoning step instead of scoring rubrics independently, improving training efficiency\. Training takes approximately 32 hours on 4 H100 GPUs\.

Process Reward Aggregation\.We first mean\-center each rubric score within its applicable subset, since some rubrics require specific preconditions such as search actions, citations, or evidence from multiple sources\. This prevents the model from favoring trajectories that only satisfy easier or more frequently applicable rubrics\. The process reward for each reasoning step is then computed as the mean of centered scores across all applicable rubrics\.

### A\.3Agentic Search Prompts

We use the following prompt templates to elicit multi\-turn agentic search\.

#### Medical QA benchmark initial turn template\.

The template used for all medical QA benchmarks with no previous turn \(search\) histories at turn 1\.

`Prompt A\.1: Medical QA initial agentic search template`

`MedCalc\-Bench template\. For MedCalc\-Bench questions, we reuse the same agentic\-search structure but replace the task\-specific search and answer instructions with the calculation variants in Prompt A\.3\. The value format instruction follows the Medmarks\-Verifiable evaluation protocol \(Warner et al\., 2026\)\. Prompt A\.2: MedCalc\-Bench template Evidence grounding prompt\. At the first turn, the model receives an empty history and no citation requirement, since no retrieved evidence has been observed\. After the search tool returns at least one real <information\> block, we append the history field and the citation rule in Prompt A\.3\. The citation rule is used only to ground the model’s internal reasoning over prior retrieval results; the model is still required to output exactly one action block\. Prompt A\.3: Evidence grounding prompt Format reminder\. When a model emits an invalid action, we provide the lightweight repair prompt in Prompt A\.3\. This prompt does not introduce new task information; it only restates the required action schema so that the next turn can be parsed by the agent environment\. Prompt A\.4: Format reminder A\.4 LLM\-judge Evaluation Table 8: Meta\-evaluation of LLM judges across five evidence\-based medicine dimensions\. Gemini 2\.5 and Gemini 3 denote Gemini\-2\.5\-Flash\-Lite and Gemini\-3\-Flash, respectively\. Agree\. denotes inter\-annotator agreement between the two judges\. Dimension Gemini 2\.5 Score Gemini 3 Score Agree\. Cohen’s κ\\kappa Ask 0\.90 0\.89 0\.93 0\.62 Acquire 0\.54 0\.57 0\.71 0\.41 Appraise 0\.48 0\.44 0\.79 0\.58 Apply 0\.41 0\.42 0\.77 0\.54 Assess 0\.19 0\.21 0\.82 0\.45 Macro Avg\. 0\.50 0\.51 0\.80 0\.52 Table 8 compares rubric scores from Gemini 2\.5 and Gemini 3\. The two judges produce nearly identical macro\-average rubric scores \(50% vs\. 51%\) with 80% average agreement, suggesting that the rubric labels are robust across judges\. Agreement is highest for Ask \(93%\) and lower for Acquire and Apply, which require more subjective assessment of search and evidence use\. The average Cohen’s κ\\kappa is 0\.52, indicating moderate agreement beyond label imbalance and supporting the reliability of our process\-quality analysis\.`