学习何时协作：通过不确定性感知路由实现选择性多智能体医学推理

# Learning When to Collaborate: Selective Multi-Agent Medical Reasoning via Uncertainty-Aware Routing
Source: [https://arxiv.org/html/2610.33544](https://arxiv.org/html/2610.33544)
Xueliang CuiAffiliation:Southern University of Science and TechnologyAffiliation:Shenzhen University of Advanced Technology, ChinaDan LuAffiliation:West China Hospital, Sichuan University, China\.[https://github\.com/JayceonHo/SMART\-Med](https://github.com/JayceonHo/SMART-Med)Ruxin Wang††thanks:Corresponding author\. Mails to rx\.wang@siat\.ac\.cn and houjc24@mails\.tsinghua\.edu\.cn\.Affiliation:Shenzhen Institutes of Advanced Technology, Chinese Academy of SciencesAffiliation:Shenzhen University of Advanced Technology, China

###### Abstract

Multi\-agent systems have emerged as a promising paradigm for multimodal medical reasoning, but existing methods rely on two assumptions that hinder practical deployment: \(i\) always\-active dense collaboration among multiple agents, and \(ii\) dependence on large cloud\-based models that raise privacy and efficiency concerns\. In this paper, we challenge both assumptions and proposeSMART\-Med, a fully local multi\-agent method that treats collaboration as a learnable decision rather than a default behavior\.*Our key insight is that not every medical query requires costly multi\-agent deliberation: many can be reliably solved by a single well\-trained agent, while only uncertain cases benefit from expert collaboration\.*Based on this insight,SMART\-Medintroduces an uncertainty\-driven routing mechanism that adaptively decides when to answer directly and when to recruit a small subset of expert agents, effectively converting dense multi\-agent reasoning into selective, query\-adaptive coordination\. To make this routing reliable, we curate a new multimodal medical datasetMedVR\-3Kwith high\-quality reasoning traces and a difficulty hierarchy, and then fine\-tune local agents using GRPO with two complementary rewards: a certainty\-aware accuracy reward and a vision\-language reward model for correctness and reasoning quality respectively\. Experiments on four medical visual question answering benchmarks show thatSMART\-Medachieves competitive performance compared to existing MAS approaches while substantially reducing inference cost \(reducing inference time by more than 3\.8×\\timesand token usage by 72%\)\. Moreover, after fine\-tuning onMedVR\-3K,SMART\-Medyields an average improvement of 6\.7% across Slake, PathVQA, VQA\-RAD, and PMC\-VQA\. Our results suggest that selective collaboration can be an efficient yet effective alternative to dense collaboration and enable practical, privacy\-preserving deployment of multi\-agent medical AI\.

## 1Introduction

Multi\-Agent Systems \(MAS\) have recently emerged as a promising paradigm for complex reasoning in various tasks[Chan et al\. \(\)](https://arxiv.org/html/2610.33544#bib.bib15);[Chen et al\. \(2023\)](https://arxiv.org/html/2610.33544#bib.bib14)\. In such systems, multiple agents, typically instantiated by Multimodal Large Language Models \(MLLMs\), are assigned specialized roles and tasks, interacting through structured communication protocols to collaboratively solve complex tasks\. Recently, this paradigm has demonstrated promising performance in Medical Visual Question Answering \(MedVQA\) and broader multimodal medical reasoning[Kim et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib7);[Tang et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib13);[Xia et al\. \(2026\)](https://arxiv.org/html/2610.33544#bib.bib17)\. Despite these advances, two critical limitations hinder the practical deployment of MAS in real\-world clinical settings\.

![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/teaser.jpeg)Figure 1:Comparison between existing MAS and our proposedSMART\-Med\. \(a\)\-\(e\) respectively depict the collaboration topology, paradigm, base model and training objective of different approaches\.First, existing MAS predominantly rely on large\-scale, cloud\-based MLLMs \(e\.g\., GPT[Korngiebel and Mooney \(2021\)](https://arxiv.org/html/2610.33544#bib.bib10), Gemini[Team \(2024\)](https://arxiv.org/html/2610.33544#bib.bib12)\), raising concerns regarding data privacy, security, and regulatory compliance\. Medical data, particularly multimodal patient records such as MRI, CT, and X\-ray scans, are sensitive, and transmitting such data to external servers is often restricted or prohibited in clinical practice\. This fundamentally limits the applicability of cloud\-dependent MAS solutions\. Second, as compared in Fig\.[1](https://arxiv.org/html/2610.33544#S1.F1)\(a\)\-\(d\), current MAS typically adopt dense communication topologies, where multiple agents engage in multi\-round interactions for every query\. While this aims to complementarily enhance reasoning quality, they introduce substantial computational overhead and inference latency, making them impractical for resource\-constrained clinical environments\. Recent studies[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.33544#bib.bib3)show that dense collaboration often yields fairly marginal gains on MedQA and MedVQA benchmarks, questioning the necessity of full\-agent participation for every query\.

To address these challenges, we proposeSMART\-Med, which operates entirely by local, small\-scale MLLMs and incorporates an adaptive collaboration mechanism\. Our key insight is that multi\-agent collaboration should be*selective rather than universal*: a certain number of queries that can be reliably solved by a single agent, while only a subset of complex cases require coordinated reasoning\.

Based on this insight,SMART\-Medintroduces two complementary technical components\. First, as depicted in Fig\.[1](https://arxiv.org/html/2610.33544#S1.F1)\(e\), we develop an uncertainty\-driven routing mechanism that prevents multi\-agent collaboration when the primary agent is certain of its response\. In particular, queries with high confidence are answered directly; only uncertain cases trigger dynamic selection of a small subset of expert agents for targeted reasoning\. This design significantly reduces inference cost and latency while preserving the benefits of collaboration when necessary\.

Second, all agents inSMART\-Medare initialized with local small\-scale MLLMs, with fine\-tuning applied to improve domain\-specific capability\. To achieve this, we first constructMedVR\-3Kbased on existing MedVQA datasets, whileMedVR\-3Kadditionally offers high\-quality reasoning chains distilled from advanced MLLMs with multi\-step filtering, and an independent judge MLLM is used to filter reasoning traces and assign difficulty scores, yielding a*hierarchical and quality\-controlled reasoning\-centric dataset*\.

Building on this, we fine\-tune the local agent using Group Relative Policy Optimization \(GRPO\) with two tailored rewards\. For answer correctness, we introduce a*certainty\-aware accuracy reward*that explicitly couples prediction confidence with correctness, encouraging the model to be confident when correct and uncertain otherwise\. For reasoning quality, we employ a vision\-language reward model, trained onMedVR\-3K, to provide fine\-grained feedback on the generated reasoning process\. Together, these two components improve both reasoning capability and uncertainty calibration, which are critical for enabling reliable selective collaboration\. Our contributions can be summarized:

- •We proposeSMART\-Med, the first medical multi\-agent system for MedVQA that operates entirely with local, small\-scale MLLMs via an uncertainty\-driven selective collaboration mechanism, dynamically triggering multi\-agent reasoning only for uncertain cases\. This design ensures data privacy, significantly reduces inference cost and latency, while preserving collaborative benefits\.
- •We constructMedVR\-3K, a hierarchical reasoning\-centric dataset\. With the aid ofMedVR\-3K, we fine\-tune the local model using GRPO with two complementary new rewards: a certainty\-aware accuracy reward that couples confidence with correctness, and a vision\-language reward model that provides fine\-grained reasoning quality feedback\.
- •Extensive experiments on four public MedVQA benchmarks \(i\.e\., Slake, PathVQA, VQA\-RAD, and PMC\-VQA\) show thatSMART\-Medachieves competitive or superior performance compared to existing multi\-agent frameworks, while significantly reducing inference cost\.

## 2Methodology

### 2\.1Problem Formulation

In MedVQA, the input consists of a medical imagex∈𝒳x\\in\\mathcal\{X\}\(e\.g\., X\-ray, CT, MRI, pathology slides\) and a natural language questionq∈𝒬q\\in\\mathcal\{Q\}\. The goal is to generate an answery∈𝒴y\\in\\mathcal\{Y\}accompanied by a reasoning chainrr\. In a multi\-agent system, we define a set ofKKagents𝒜=\{A1,…,AK\}\\mathcal\{A\}=\\\{A\_\{1\},\\dots,A\_\{K\}\\\}, where each agent is instantiated by a MLLM with parametersθk\\theta\_\{k\}\. Agents may specialize in different reasoning styles or roles \(e\.g\., visual grounding, clinical knowledge, verification\)\. Current MAS generally produce answers following a dense collaboration protocol:

y=ℱdense​\(x,q,𝒜\),y=\\mathcal\{F\}\_\{\\text\{dense\}\}\(x,q;\\mathcal\{A\}\),\(1\)where multiple agents engage in multi\-round communication\. In contrast, we aim to learn a routing policyπθ\\pi\_\{\\theta\}that smartly switches between single\-agent and multi\-agent modes:

y=ℱsparse​\(x,q,𝒜,πθ\),y=\\mathcal\{F\}\_\{\\text\{sparse\}\}\(x,q;\\mathcal\{A\},\\pi\_\{\\theta\}\),\(2\)whereπθ\\pi\_\{\\theta\}determines whether multi\-agent collaboration is needed and, if so, how many agents should be activated, conditioned on the input\. This transforms the dense communication topology of a MAS into a sparse, query\-adaptive structure by activating only a few \(or even a single\) agent\.

![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/overview.jpg)Figure 2:Overview ofSMART\-Med\. The framework unifies reasoning\-centric training and uncertainty\-aware routing in a closed loop:MedVR\-3Kand reward\-guided optimization improve both reasoning capability and uncertainty calibration, while the learned router adaptively decides whether to answer directly or trigger selective expert collaboration\.
### 2\.2SMART\-Med: Learning When to Collaborate

In real\-world clinical practice, the first step when a patient visits a hospital is triage, assigning the patient to an appropriate department based on symptoms\. InSMART\-Med, we employ a router agent that plays the role of a general practitioner\. Its parameters are our optimization targetπθ\\pi\_\{\\theta\}, which decides when to invoke multi\-agent collaboration\.

Concretely, we instantiate a local MLLM as the router\. It generates an initial assessmenty~\\tilde\{y\}and appends a certainty token aftery~\\tilde\{y\}for each medical query:

y~∈𝒴∪\{CERTAIN∣UNCERTAIN\},\\tilde\{y\}\\in\\mathcal\{Y\}\\cup\\\{\\texttt\{CERTAIN\}\\mid\\texttt\{UNCERTAIN\}\\\},\(3\)where the certainty token indicates whether the router is sufficiently certain about its initial assessmenty~\\tilde\{y\}\. As shown in Fig\.[2](https://arxiv.org/html/2610.33544#S2.F2), if the router is confident \(outputsCERTAIN\),SMART\-Meddirectly adoptsy~\\tilde\{y\}as the final answer and skips the expensive multi\-agent collaboration\. Otherwise, the router recruits a small subset of expert agents to perform collaborative reasoning\. In this case, a moderator agent gathers the analysis reports from the recruited experts and makes the final decision\. This discussion\-and\-decision workflow follows prior work[Tang et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib13);[Xia et al\. \(2026\)](https://arxiv.org/html/2610.33544#bib.bib17), with the designed prompts and role descriptions provided in Appendix[E](https://arxiv.org/html/2610.33544#A5)\. Thus, the workflow ofSMART\-Medis fully adaptive, operating in either single\-agent or multi\-agent mode depending on the given query\.

#### Reward\-Shaped Optimization\.

We employ Group Relative Policy Optimization \(GRPO\)[Shao et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib4)to jointly optimize answer correctness, uncertainty calibration, reasoning quality, and output format compliance\. First, we define a format reward:

RFMT​\(oi\)=𝕀⁡\[oi​strictly matches the required output schema\],R\_\{\\mathrm\{FMT\}\}\(o\_\{i\}\)=\\mathbb\{I\}\\\!\\left\[o\_\{i\}\\text\{ strictly matches the required output schema\}\\right\],\(4\)which enforces structural constraints \(e\.g\., certainty level, reasoning chain, final answer\) on the generated responseoio\_\{i\}\.

Next, we introduce two customized rewards: a*Certainty\-aware Accuracy Reward \(CAR\)*and a*Reasoning\-quality Reward \(RR\)*\. Letyiy\_\{i\}be the answer extracted fromoio\_\{i\},cic\_\{i\}the certainty decision \(ci=1c\_\{i\}=1forCERTAIN,00forUNCERTAIN\), andy⋆y^\{\\star\}the ground\-truth answer\. We define CAR as:

RCAR​\(oi\)=𝕀⁡\(yi=y⋆\)⋅\[ci\+\(1−ci\)​α\]−𝕀⁡\(yi≠y⋆\)⋅ci​β,R\_\{\\mathrm\{CAR\}\}\(o\_\{i\}\)=\\mathbb\{I\}\(y\_\{i\}=y^\{\\star\}\)\\cdot\\big\[c\_\{i\}\+\(1\-c\_\{i\}\)\\alpha\\big\]\-\\mathbb\{I\}\(y\_\{i\}\\neq y^\{\\star\}\)\\cdot c\_\{i\}\\beta,\(5\)whereα∈\(0,1\)\\alpha\\in\(0,1\)gives reduced credit to correct but uncertain outputs, andβ\>0\\beta\>0penalizes confident mistakes\. This reward induces a selective prediction objective with a*reject option*, encouraging the model to outputUNCERTAINwhen its direct answer is likely wrong\. For RR, we use a frozen Vision\-Language Model \(VLM\) fine\-tuned on our dataset:

RRR​\(oi\)=fVLM​\(x,q,ri,yi\),R\_\{\\mathrm\{RR\}\}\(o\_\{i\}\)=f\_\{\\mathrm\{VLM\}\}\(x,q,r\_\{i\},y\_\{i\}\),\(6\)whererir\_\{i\}is the generated reasoning trace\. The combined raw reward is:

R¯i=λfmt​RFMT​\(oi\)\+λcar​RCAR​\(oi\)\+λrr​RRR​\(oi\),\\bar\{R\}\_\{i\}=\\lambda\_\{\\mathrm\{fmt\}\}R\_\{\\mathrm\{FMT\}\}\(o\_\{i\}\)\+\\lambda\_\{\\mathrm\{car\}\}R\_\{\\mathrm\{CAR\}\}\(o\_\{i\}\)\+\\lambda\_\{\\mathrm\{rr\}\}R\_\{\\mathrm\{RR\}\}\(o\_\{i\}\),\(7\)with coefficientsλfmt,λcar,λrr\\lambda\_\{\\mathrm\{fmt\}\},\\lambda\_\{\\mathrm\{car\}\},\\lambda\_\{\\mathrm\{rr\}\}controlling the relative importance of format compliance, answer correctness/certainty, and reasoning quality\.

Following standard GRPO, we sample a group ofGGcandidate outputs\{oi\}i=1G\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}for each input\(x,q\)\(x,q\)and standardize the rewards within the group:

Ai=R¯i−μRσR,A\_\{i\}=\\frac\{\\bar\{R\}\_\{i\}\-\\mu\_\{R\}\}\{\\sigma\_\{R\}\},\(8\)whereμR\\mu\_\{R\}andσR\\sigma\_\{R\}are the group mean and variance\. The standardized advantageAiA\_\{i\}measures how much better a response is relative to its group under our rewards\. For simplicity, we omit the KL constraint, and the objective can be expressed in an approximate form as follows:

−𝔼\(x,q\)∼𝒟,\{oi\}i=1G∼πθold​\[1G​∑i=1G1\|oi\|​∑t=1\|oi\|min⁡\(ri,t​\(θ\)​Ai,t,clip⁡\(ri,t​\(θ\),1−ϵ,1\+ϵ\)​Ai,t\)\]\\displaystyle\-\\mathbb\{E\}\_\{\(x,q\)\\sim\\mathcal\{D\},\\,\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{t=1\}^\{\|o\_\{i\}\|\}\\min\\Big\(r\_\{i,t\}\(\\theta\)A\_\{i,t\},\\operatorname\{clip\}\\big\(r\_\{i,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\big\)A\_\{i,t\}\\Big\)\\Bigg\]\(9\)\\displaystyle,whereπref\\pi\_\{\\mathrm\{ref\}\}is a frozen reference policy, andβKL\\beta\_\{\\mathrm\{KL\}\}controls the deviation from the reference\. This objective jointly encourages correct answers, calibrated uncertainty, high\-quality reasoning traces, and strict format adherence\.

#### Theoretical Justification\.

We provide a decision\-theoretic interpretation of our certainty\-aware design, together with its implications for optimization under GRPO\. Letz∈\{0,1\}z\\in\\\{0,1\\\}denote whether the router’s direct answer is correct, and let

p⁡\(x,q\)=Pr⁡\(z=1∣x,q\)p\(x,q\)=\\Pr\(z=1\\mid x,q\)\(10\)denote the conditional correctness probability of the direct answer\. In the standard setting without deferral, the model always produces an answer, yielding the expected risk under00\-11loss:

Rno\_rej​\(x,q\)=1−p⁡\(x,q\)\.R\_\{\\text\{no\\\_rej\}\}\(x,q\)=1\-p\(x,q\)\.\(11\)
Our formulation introduces an additional decision: the router may either answer directly or declareUNCERTAIN, which triggers multi\-agent collaboration\. This can be interpreted as a form of*deferred decision\-making*, where uncertain cases are routed to a collaborative reasoning process\. To characterize the reward\-induced routing preference, consider an idealized decision setting in whichp⁡\(x,q\)p\(x,q\)is calibrated and the router selects the action maximizing its expected certainty\-aware reward\. Under the certainty\-aware reward in Eq\. \([5](https://arxiv.org/html/2610.33544#S2.E5)\), the expected utilities of the two actions can be written as

Uans=p⁡\(x,q\)−\(1−p⁡\(x,q\)\)​β,Udef=p⁡\(x,q\)​α,U\_\{\\text\{ans\}\}=p\(x,q\)\-\(1\-p\(x,q\)\)\\beta,\\qquad U\_\{\\text\{def\}\}=p\(x,q\)\\alpha,\(12\)whereα∈\(0,1\)\\alpha\\in\(0,1\)gives partial credit to correct but deferred outputs, andβ\>0\\beta\>0penalizes overconfident errors\. Comparing the two utilities gives the reward\-induced Bayes\-optimal threshold:

Uans≥Udef⟺p\(x,q\)≥τBayes,τBayes=β1−α\+β\.U\_\{\\text\{ans\}\}\\geq U\_\{\\text\{def\}\}\\quad\\Longleftrightarrow\\quad p\(x,q\)\\geq\\tau\_\{\\mathrm\{Bayes\}\},\\qquad\\tau\_\{\\mathrm\{Bayes\}\}=\\frac\{\\beta\}\{1\-\\alpha\+\\beta\}\.\(13\)Thus, the certainty\-aware reward induces a non\-trivial utility boundary: direct answering is preferred when the estimated correctness probability exceedsτBayes\\tau\_\{\\mathrm\{Bayes\}\}, whereas deferral is preferred otherwise\. The detailed derivation of this reward\-induced threshold is provided in Appendix[B](https://arxiv.org/html/2610.33544#A2)\. Importantly,τBayes\\tau\_\{\\mathrm\{Bayes\}\}characterizes the*reward\-induced theoretical threshold*rather than the exact decision boundary of the GRPO\-trained router\. In practice, GRPO learns a parameterized policyπθ​\(a∣x,q\)\\pi\_\{\\theta\}\(a\\mid x,q\), and the optimized router is given by

θ⋆≈arg⁡maxθ​JGRPO​\(θ,α,β\),\\theta^\{\\star\}\\approx\\arg\\max\_\{\\theta\}J\_\{\\mathrm\{GRPO\}\}\(\\theta;\\alpha,\\beta\),\(14\)with the actual routing decision determined by

aθ⋆​\(x,q\)=arg⁡maxa∈\{ANS,DEF\}​πθ⋆​\(a∣x,q\)\.a\_\{\\theta^\{\\star\}\}\(x,q\)=\\arg\\max\_\{a\\in\\\{\\mathrm\{ANS\},\\mathrm\{DEF\}\\\}\}\\pi\_\{\\theta^\{\\star\}\}\(a\\mid x,q\)\.\(15\)Accordingly, the actual learned routing boundary is implicitly defined by

πθ⋆​\(ANS∣x,q\)=πθ⋆​\(DEF∣x,q\)\.\\pi\_\{\\theta^\{\\star\}\}\(\\mathrm\{ANS\}\\mid x,q\)=\\pi\_\{\\theta^\{\\star\}\}\(\\mathrm\{DEF\}\\mid x,q\)\.\(16\)The derivation above provides a principled decision\-theoretic interpretation of howα\\alphaandβ\\betashape the relative utility of direct answering and deferral\. In practice, the resulting routing behavior is learned implicitly through GRPO and depends not only onα\\alphaandβ\\beta, but also on the training data, optimization dynamics, and the learned policy\.

From an optimization perspective, a standard rewardRbase=𝕀⁡\(y=y⋆\)\+λfmt​𝕀​\(format\)R\_\{\\text\{base\}\}=\\mathbb\{I\}\(y=y^\{\\star\}\)\+\\lambda\_\{\\text\{fmt\}\}\\mathbb\{I\}\(\\text\{format\}\)cannot distinguish overconfident errors from calibrated deferral, since both are treated identically with respect to answer correctness, which optimizes in a more narrow landscape and cannot penalize over\-confident or over\-conservative cases\.

![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/statistics.png)Figure 3:Statistics of our constructedMedVR\-3K\. \(a\)\-\(d\) respectively present the average quality score, distribution of difficulty level, response length and quality assessment length\.

### 2\.3MedVR\-3K: Medical Reasoning\-Centric Dataset

To improve the reasoning capability of local MLLMs, we constructMedVR\-3K, a reasoning\-centric multimodal medical dataset\. As shown in Fig\.[3](https://arxiv.org/html/2610.33544#S2.F3), besides the medical question and image \(e\.g\., MRI, CT, pathology\), each sample inMedVR\-3Kadditionally comprises \(i\) a gold response with a step\-by\-step reasoning chain, \(ii\) a difficulty label, and \(iii\) a quality score along with a justification\. Unlike prior MAS that only supervise final answers[Xia et al\. \(2026\)](https://arxiv.org/html/2610.33544#bib.bib17),MedVR\-3Kexplicitly provides the reasoning process, offering a more informative supervision signal for multimodal reasoning\. Examples are shown in Appendix[D](https://arxiv.org/html/2610.33544#A4)\.

#### Data Collection\.

As illustrated in Fig\.[2](https://arxiv.org/html/2610.33544#S2.F2), we collect question\-image pairs from the training splits of four public benchmarks: Slake[Liu et al\. \(2021\)](https://arxiv.org/html/2610.33544#bib.bib25), PMC\-VQA[Zhang et al\. \(2024b\)](https://arxiv.org/html/2610.33544#bib.bib18), VQA\-RAD[Lau et al\. \(2018\)](https://arxiv.org/html/2610.33544#bib.bib26), and PathVQA[He et al\. \(2020\)](https://arxiv.org/html/2610.33544#bib.bib9)\. For each sample\(x,q\)\(x,q\), we query a set of strong candidate MLLMs \(i\.e\., GPT\-5\.4[OpenAI \(2024\)](https://arxiv.org/html/2610.33544#bib.bib11), Gemini 3 Pro[Team \(2024\)](https://arxiv.org/html/2610.33544#bib.bib12), Qwen3\-Max[Bai et al\. \(2025\)](https://arxiv.org/html/2610.33544#bib.bib23), and Kimi K2\.5[Team et al\. \(2025\)](https://arxiv.org/html/2610.33544#bib.bib16)\) and require each response to include a structured, step\-by\-step reasoning chain together with a final answer\. Among the candidate responses, we retain only those whose final answers match the ground\-truth label\. This first\-stage filtering ensures that the dataset contains reliable reasoning traces instead of fluent but incorrect explanations\.

#### Annotation and Quality Control\.

After correctness filtering, we perform a second\-stage quality control using GPT\-5\.4 as an independent judgeJJ\. Instead of a single scalar score, the judge evaluates each response along three dimensions:*Logic Consistency*,*Factual Correctness*, and*Image Alignment*\. Formally, for a candidate responserr, the judge produces a score vector

𝐬=\(slogic,sfact,simg\)=J⁡\(x,q,r\),\\mathbf\{s\}=\\big\(s\_\{\\mathrm\{logic\}\},\\,s\_\{\\mathrm\{fact\}\},\\,s\_\{\\mathrm\{img\}\}\\big\)=J\(x,q,r\),\(17\)where the scoressis in range of\[1,5\]\[1,5\]\. We adopt a dimension\-wise filtering strategy and retain a response only if it satisfies all quality thresholds \(denoted byγ\\gamma\):

slogic≥γlogic,sfact≥γfact,simg≥γimg\.s\_\{\\mathrm\{logic\}\}\\geq\\gamma\_\{\\mathrm\{logic\}\},\\quad s\_\{\\mathrm\{fact\}\}\\geq\\gamma\_\{\\mathrm\{fact\}\},\\quad s\_\{\\mathrm\{img\}\}\\geq\\gamma\_\{\\mathrm\{img\}\}\.\(18\)This multi\-criteria filtering ensures that retained reasoning traces are not only correct in final answers but also logically coherent, factually accurate, and well\-grounded in visual evidence\. Compared to single\-score filtering, this approach avoids cases where strong performance in one dimension compensates for deficiencies in others, yielding more reliable supervision\. Subsequently, as shown in Fig\.[3](https://arxiv.org/html/2610.33544#S2.F3), the average quality score ofMedVR\-3Kreaches around 3\.60\.

#### Difficulty Annotation\.

Beyond quality control, we further leverage the judge model to annotate the difficulty of each sample\. Specifically, the judge assigns a difficulty labelddbased on both question complexity and visual complexity:

d=J⁡\(x,q,r\)∈\{basic,medium,hard\}\.d=J\(x,q,r\)\\in\\\{\\texttt\{basic\},\\texttt\{medium\},\\texttt\{hard\}\\\}\.\(19\)The difficulty is determined by factors such as the number of reasoning steps, the level of domain knowledge required, and the ambiguity or complexity of visual evidence\. In particular,basicsamples typically involve direct visual recognition, whilemediumandhardsamples require multi\-step reasoning, cross\-region inference, or integration of domain\-specific knowledge\. This hierarchical annotation enables curriculum learning, allowing the model to progressively learn from simple to complex reasoning patterns\. Detailed prompts are provided in Appendix[E](https://arxiv.org/html/2610.33544#A5)\.

After the two\-stage filtering and hierarchical annotation,MedVR\-3Kcontains 3,477 multimodal medical QA samples with structured reasoning chains, which serves as a hierarchical MedVQA dataset specifically designed for reasoning\-centric multimodal medical learning\.

## 3Experiments

### 3\.1Setup

We evaluate our method on four widely\-used public MedVQA datasets: Slake[Liu et al\. \(2021\)](https://arxiv.org/html/2610.33544#bib.bib25), PathVQA[He et al\. \(2020\)](https://arxiv.org/html/2610.33544#bib.bib9), VQA\-RAD[Lau et al\. \(2018\)](https://arxiv.org/html/2610.33544#bib.bib26), PMC\-VQA[Zhang et al\. \(2024b\)](https://arxiv.org/html/2610.33544#bib.bib18), which cover diverse imaging modalities and clinical scenarios\. We compare our method with a series of single agent and multi\-agent methods as follows:

- •Single Agent\.We adopt different types and sizes of representative open\-source MLLMs to initialize a single agent with basic prompts to serve as a reference, including Qwen3\-VL series[Bai et al\. \(2025\)](https://arxiv.org/html/2610.33544#bib.bib23), LLaVA[Liu et al\. \(2023\)](https://arxiv.org/html/2610.33544#bib.bib21), Ministral 3 series[Liu et al\. \(2026\)](https://arxiv.org/html/2610.33544#bib.bib22), and the specialized medical models, the BiomedGPT[Zhang et al\. \(2024a\)](https://arxiv.org/html/2610.33544#bib.bib6), MedFlamingo[Moor et al\. \(2023\)](https://arxiv.org/html/2610.33544#bib.bib1), HuatuoGPT\-Vision[Chen et al\. \(2024a\)](https://arxiv.org/html/2610.33544#bib.bib2), and LLaVA\-Med[Li et al\. \(2023\)](https://arxiv.org/html/2610.33544#bib.bib5)\.
- •Multi\-agent Collaboration\.We also implement the latest and state\-of\-the\-art MAS methods: MDAgents[Kim et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib7), MedAgent[Tang et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib13), MOA[Wang et al\. \(2024\)](https://arxiv.org/html/2610.33544#bib.bib20), SMOA[Li et al\. \(2025\)](https://arxiv.org/html/2610.33544#bib.bib19), Reconcile[Chen et al\. \(2024b\)](https://arxiv.org/html/2610.33544#bib.bib8), and ColaCare[Wang et al\. \(2025\)](https://arxiv.org/html/2610.33544#bib.bib24)\.

More information about the datasets, implementation details and evaluation metrics can be found in Appendix[A](https://arxiv.org/html/2610.33544#A1)\.

Table 1:Accuracy comparison with state\-of\-the\-art baseline models on Slake, PathVQA, VQA\-RAD and PMC\-VQA datasets under multi\-choice and free\-form questions\.CategoryMethodsMultiple\-Choice Questions \(MC\)Free\-form Questions \(FF\)SlakePathVQAVQA\-RADPMC\-VQASlakePathVQAVQA\-RADSingle agent\(Specialized model\)BiomedGPT53\.5±\\pm0\.3 57\.0±\\pm0\.7 52\.0±\\pm0\.2 41\.5±\\pm0\.7 54\.7±\\pm0\.8 39\.5±\\pm2\.2 54\.8±\\pm3\.4 Med\-Flamingo61\.5±\\pm1\.5 58\.5±\\pm3\.0 54\.5±\\pm1\.9 38\.0±\\pm2\.0 57\.9±\\pm1\.1 34\.7±\\pm2\.8 51\.5±\\pm4\.2 LLaVA\-Med59\.0±\\pm0\.1 52\.5±\\pm0\.1 60\.5±\\pm0\.4 36\.0±\\pm1\.3 59\.6±\\pm0\.4 48\.5±\\pm0\.5 65\.8±\\pm1\.7 HuatuoGPT\-Vision69\.5±\\pm1\.8 59\.0±\\pm1\.4 64\.0±\\pm1\.6 41\.0±\\pm2\.2 59\.6±\\pm1\.8 45\.9±\\pm1\.6 63\.8±\\pm0\.6 Single agent\(Local models\)LLaVA\-v1\.6\-7B65\.0±\\pm0\.5 52\.0±\\pm1\.2 54\.0±\\pm0\.7 32\.0±\\pm2\.1 52\.4±\\pm2\.8 31\.3±\\pm0\.7 50\.8±\\pm2\.9 Minstral\-3\-3B\-Instruct68\.0±\\pm1\.2 51\.0±\\pm0\.4 55\.5±\\pm1\.1 45\.0±\\pm2\.3 67\.2±\\pm1\.2 33\.3±\\pm1\.9 45\.7±\\pm0\.3 Minstral\-3\-8B\-Instruct70\.0±\\pm0\.8 63\.0±\\pm1\.7 57\.0±\\pm1\.5 38\.5±\\pm1\.1 77\.3±\\pm1\.0 41\.7±\\pm0\.3 62\.9±\\pm0\.7 Qwen3\-VL\-2B\-Instruct76\.0±\\pm0\.5 58\.5±\\pm1\.1 66\.7±\\pm1\.0 30\.2±\\pm1\.3 73\.6±\\pm2\.2 40\.5±\\pm1\.6 58\.0±\\pm0\.1 Qwen3\-VL\-4B\-Instruct76\.1±\\pm1\.4 64\.6±\\pm2\.1 70\.8±\\pm1\.8 35\.3±\\pm2\.8 75\.4±\\pm0\.8 44\.3±\\pm2\.5 62\.6±\\pm4\.2 Qwen3\-VL\-8B\-Instruct71\.9±\\pm2\.3 59\.0±\\pm0\.4 64\.2±\\pm0\.5 50\.2±\\pm2\.4 73\.3±\\pm0\.7 45\.8±\\pm3\.0 60\.3±\\pm5\.0 Qwen3\-VL\-8B\-Thinking73\.6±\\pm0\.8 59\.0±\\pm1\.0 60\.5±\\pm2\.3 33\.8±\\pm1\.0 67\.6±\\pm2\.4 49\.6±\\pm0\.8 53\.6±\\pm1\.8 Multi\-agent\(Local models\)MedAgents72\.0±\\pm1\.4 73\.7±\\pm2\.2 72\.0±\\pm0\.1 52\.5±\\pm0\.1 74\.0±\\pm0\.8 48\.8±\\pm1\.6 67\.0±\\pm3\.0 MDAgents70\.2±\\pm1\.2 75\.6±\\pm1\.0 68\.6±\\pm2\.0 50\.6±\\pm1\.0 69\.1±\\pm1\.9 46\.6±\\pm0\.6 57\.4±\\pm4\.3 MOA79\.0±\\pm1\.0 69\.2±\\pm2\.1 71\.5±\\pm0\.5 51\.5±\\pm1\.1 72\.9±\\pm0\.7 52\.6±\\pm4\.0 62\.3±\\pm2\.7 Reconcile72\.8±\\pm1\.0 64\.2±\\pm1\.2 66\.0±\\pm0\.5 49\.0±\\pm1\.0 58\.1±\\pm1\.0 43\.1±\\pm1\.8 57\.5±\\pm4\.3 SMOA68\.0±\\pm1\.1 62\.2±\\pm3\.7 63\.6±\\pm1\.2 49\.3±\\pm2\.6 60\.3±\\pm1\.0 38\.9±\\pm1\.1 63\.0±\\pm1\.5 ColaCare71\.3±\\pm1\.5 60\.8±\\pm2\.0 65\.3±\\pm0\.6 50\.0±\\pm0\.8 60\.5±\\pm1\.8 45\.2±\\pm1\.0 53\.8±\\pm2\.6 Adaptive agent\(Local models\)SMART\-Med\(Ours\)76\.7±\\pm1\.0 68\.6±\\pm1\.3 72\.5±\\pm0\.5 48\.5±\\pm2\.1 78\.2±\\pm1\.3 48\.7±\\pm0\.2 64\.1±\\pm3\.2 SMART\-Med\* \(Ours\)87\.0±\\pm0\.577\.0±\\pm1\.174\.0±\\pm1\.672\.5±\\pm1\.084\.3±\\pm2\.558\.5±\\pm3\.569\.4±\\pm3\.7

Note:MC: Multiple\-choice question answering; FF: Free\-form \(open\-ended\) question answering\.Boldindicates the best performance, andUnderlinedindicates the second\-best performance per column \(dataset and task\)\. All scores are reported as mean±\\pmstandard deviation

by testing 5 times independently\. For fair comparison, we use Qwen3\-VL\-4B\-Instruct as the base model for MAS approaches\.SMART\-Med\* denotes our method after fine\-tuning onMedVR\-3K\.
### 3\.2Results Analysis

#### Quantitative Comparison\.

As shown in Table[1](https://arxiv.org/html/2610.33544#S3.T1), without any fine\-tuning,SMART\-Medalready achieves competitive performance among local multi\-agent approaches\. For example, on FF tasks,SMART\-Medattains 78\.2% on Slake, outperforming the best local baseline \(77\.3% from Minstral\-3\-8B\-Instruct\) and surpassing most multi\-agent methods\. On PathVQA and VQA\-RAD, it achieves 48\.7% and 64\.1%, respectively, remaining competitive with dense multi\-agent frameworks such as MOA \(52\.6%, 62\.3%\) and MedAgents \(48\.8%, 67\.0%\)\. On MC tasks,SMART\-Medalso performs strongly, reaching 76\.7% on Slake and achieving the second\-best result on VQA\-RAD \(72\.5%\), while maintaining reasonable performance on the more challenging PMC\-VQA \(48\.5%\)\.

After fine\-tuning onMedVR\-3K,SMART\-Med\* further achieves substantial improvements across all benchmarks\. On MC tasks,SMART\-Med\* reaches 87\.0% on Slake, 77\.0% on PathVQA, 74\.0% on VQA\-RAD, and 72\.5% on PMC\-VQA, exceeding the best baselines by \+8\.0%, \+1\.4%, \+2\.0%, and \+20\.0% points, respectively\. On FF tasks, it achieves 84\.3% on Slake, 58\.5% on PathVQA, and 69\.4% on VQA\-RAD, establishing new state\-of\-the\-art results among local models\. Notably, the largest gains are observed on PMC\-VQA and PathVQA, which involve more complex reasoning, highlighting the effectiveness of our reasoning\-centric training and reward design\. Overall, these results show that whileSMART\-Medprovides a strong and efficient zero\-shot solution,SMART\-Med\* further unlocks its potential through reasoning supervision and uncertainty\-aware optimization, leading to consistently superior performance\.

To disentangle the accuracy gains ofSMART\-Med\* from the effect of fine\-tuning onMedVR\-3Kitself, we additionally fine\-tune two representative baselines, MOA and MDAgent, with the identicalMedVR\-3Kcorpus and training pipeline \(replacing the certainty\-aware reward with the standard accuracy reward, since neither models a certainty token\)\. Tableshows thatMedVR\-3Kconsistently benefits all methods, yetSMART\-Med\* still achieves the best overall performance after controlling for identical post\-training, indicating that uncertainty\-aware routing provides complementary benefits beyond domain\-specific fine\-tuning alone\.

Table 2:Reasoning quality comparison\. LC, FC, and IA respectively represent the Logic Consistency, Factual Correctness and Image Alignment\.ScoreMDAgentMedAgentSMOAMOAColaCareSMART\-MedSMART\-Med\*LC2\.893\.313\.483\.423\.283\.243\.55FC2\.803\.113\.373\.123\.202\.963\.38IA2\.563\.193\.293\.403\.183\.143\.57Avg\.2\.753\.203\.383\.313\.223\.113\.50

#### Qualitative Comparison\.

We randomly select 100 samples, covering both FF and MC questions for reasoning quality comparison\. Similarly, we employ the GPT\-5\.4 as the judge model, and we evaluate the generated reasoning along three dimensions: Logic Consistency \(LC\), Factual Correctness \(FC\), and Image Alignment \(IA\)\. The prompts for the judge model can be found in Appendix[E](https://arxiv.org/html/2610.33544#A5)\. The results are reported in Table[2](https://arxiv.org/html/2610.33544#S3.T2)\. It can be seen thatSMART\-Med\* achieves the best performance across all dimensions, with an average score of 3\.50, outperforming prior multi\-agent methods such as SMOA \(3\.38\) and MOA \(3\.31\)\. In particular,SMART\-Med\* shows strong improvements in Image Alignment \(3\.57 vs\. 3\.40 of MOA\) and Logic Consistency \(3\.55 vs\. 3\.48 of SMOA\), indicating that our method generates reasoning that is better grounded in visual evidence while maintaining coherent logical structure\. ComparingSMART\-MedandSMART\-Med\*, we observe consistent gains across all metrics \(Avg\.: 3\.11→\\rightarrow3\.50\), validating the effectiveness of our GRPO training with certainty\-aware and reasoning rewards\. More validations and analysis can be found in Appendix[C](https://arxiv.org/html/2610.33544#A3)\.

![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/efficiency.png)Figure 4:Efficiency comparison ofSMART\-Medand other MAS methods\. We compare the average runtime \(second\), average I/O token usage and the average model calls \(times\) per question\.
#### Efficiency Analysis\.

We evaluate efficiency of existing MAS approaches in terms of latency, token usage, and communication overhead on 100 randomly\-sampled questions\. From Fig\.[4](https://arxiv.org/html/2610.33544#S3.F4), it has been shown thatSMART\-Medis significantly more efficient than other approaches, requiring less than 10 seconds for every query on average, corresponding to a 3\.8×\\timesspeedup over the fastest baseline \(i\.e\., MDAgents\) and up to 11\.9×\\timesover dense collaboration methods \(i\.e\., ColaCare\)\. In addition,SMART\-Medmerely consumes around 600 input tokens and 100 output tokens per query, reducing communication cost by over 72% and 88%, respectively\. It also requires only 1\.17 model calls per question, far less than 5\.51–17\.92 in prior methods\. These results verify that selective, uncertainty\-aware routing substantially reduces computation and communication overhead\.

Table 3:Ablation study results on MC and FF questions\.ModelMCFFSlakePathVQAVQA\-RADPMC\-VQASlakePathVQAVQA\-RADSingle agent76\.1±\\pm1\.4 64\.6±\\pm2\.1 70\.8±\\pm1\.8 35\.3±\\pm2\.8 75\.4±\\pm0\.8 44\.3±\\pm2\.5 62\.6±\\pm4\.2 Multi\-agent\+ Dense collaboration78\.5±\\pm0\.3 66\.7±\\pm2\.4 71\.3±\\pm2\.2 50\.1±\\pm0\.5 75\.7±\\pm2\.8 51\.2±\\pm1\.7 64\.5±\\pm1\.7 \+ Uncertainty routing76\.7±\\pm1\.0 68\.6±\\pm1\.3 72\.5±\\pm0\.5 48\.5±\\pm2\.1 78\.2±\\pm1\.3 48\.7±\\pm0\.2 64\.1±\\pm3\.2 \+ Certain\. reward85\.5±\\pm1\.0 75\.0±\\pm0\.9 73\.1±\\pm1\.2 69\.5±\\pm1\.1 82\.2±\\pm1\.8 56\.8±\\pm1\.2 68\.7±\\pm2\.6 \+ Reason\. reward87\.0±\\pm0\.577\.0±\\pm1\.174\.0±\\pm1\.672\.5±\\pm1\.084\.3±\\pm2\.558\.5±\\pm3\.569\.4±\\pm3\.7

### 3\.3Ablation Study

#### Effect of Agent Collaboration\.

From Table[3](https://arxiv.org/html/2610.33544#S3.T3), the dense collaboration improves MC accuracy on PMC\-VQA from 35\.3% to 50\.1% \(\+14\.8%\) and PathVQA from 64\.6% to 66\.7%, while also bringing gains on FF tasks such as PathVQA \(44\.3% to 51\.2%\)\. Replacing dense collaboration with uncertainty\-driven routing achieves comparable or even better performance without any additional training\. Notably, uncertainty routing outperforms dense collaboration on PathVQA \(66\.7% to 68\.6%\) and VQA\-RAD \(71\.3% to 72\.5%\) in MC, and achieves higher FF accuracy on Slake \(75\.7% to 78\.2%\)\. These results confirm that multi\-agent collaboration should be*selective rather than universal*: many queries can be reliably solved by a single agent, while only a subset of complex cases benefit from the agent collaboration\.

#### Effect of Agent Learning\.

Training with GRPO using our proposed Certainty\-Aware Accuracy Reward \(CAR\) further boosts performance across all benchmark\. For instance, MC accuracy on PMC\-VQA improves significantly from 48\.5% to 69\.5%, and PathVQA from 68\.6% to 75\.0%, while FF performance on Slake increases from 78\.2% to 82\.2%\. Furthermore, incorporating the Reasoning\-quality Reward \(RR\) leads to additional gains, particularly on FF tasks where reasoning quality is critical \(e\.g\., PathVQA: 56\.8% to 58\.5%, Slake: 82\.2% to 84\.3%\), and also improves MC performance \(e\.g\., PMC\-VQA: 69\.5% to 72\.5%\)\. These results show that the combination of CAR and RR enables the model to both make calibrated routing decisions and generate higher\-quality reasoning, resulting in the best overall performance\.

## 4Conclusion

In this paper, we presentSMART\-Med, an adaptive and efficient multi\-agent framework for medical multimodal reasoning that leverages uncertainty\-driven sparse routing to enable selective collaboration\. To support effective training, we constructMedVR\-3K, a reasoning\-centric MedVQA benchmark with hierarchical difficulty, and propose a unified reward design that jointly models correctness, uncertainty, reasoning quality, and structural consistency\. Experiments on multiple benchmarks show thatSMART\-Medachieves state\-of\-the\-art performance among local models while significantly reducing computational overhead and preserving data privacy\. These results highlight the potential of fully local agent systems with reasoning\-centric fine\-tuning for building practical, reliable, and deployable medical AI systems\.

## Acknowledgment

This paper was supported in part by the National Key R&D Program of China under Grant 2022YFA1008300, in part by the National Natural Science Foundation of China under Grant 12471308, and in part by the Excellent Young Scholars of Shenzhen under Grant RCYX20231211090247060\.

## References

- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[Appendix A](https://arxiv.org/html/2610.33544#A1.SS0.SSS0.Px1.p1.1),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- \[2\]C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. LiuChatEval: towards better llm\-based evaluators through multi\-agent debate\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p1.1)\.
- Chenet al\.\(2023\)H\. Chen, W\. Ji, L\. Xu, and S\. ZhaoMulti\-agent consensus seeking via large language models\.arXiv preprint arXiv:2310\.20151\.Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p1.1)\.
- Chenet al\.\(2024a\)J\. Chen, R\. Ouyang, A\. Gao, S\. Chen, G\. H\. Chen, X\. Wang, R\. Zhang, Z\. Cai, K\. Ji, G\. Yu, X\. Wan, and B\. WangHuatuoGPT\-vision, towards injecting medical visual knowledge into multimodal llms at scale\.arXiv\.External Links:2406\.19280,[Link](https://arxiv.org/abs/2406.19280)Cited by:[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- Chenet al\.\(2024b\)J\. Chen, S\. Saha, and M\. BansalReconcile: round\-table conference improves reasoning via consensus among diverse llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7066–7085\.Cited by:[2nd item](https://arxiv.org/html/2610.33544#S3.I1.i2.p1.1)\.
- Heet al\.\(2020\)X\. He, Y\. Zhang, L\. Mou, E\. Xing, and P\. XiePathVQA: 30000\+ questions for medical visual question answering\.arXiv\.Cited by:[2nd item](https://arxiv.org/html/2610.33544#A1.I1.i2.p1.1),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.33544#S3.SS1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.International Conference on Learning Representation1\(2\),pp\. 3\.Cited by:[Appendix A](https://arxiv.org/html/2610.33544#A1.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2024\)Y\. Kim, C\. Park, H\. Jeong, Y\. S\. Chan, X\. Xu, D\. McDuff, H\. Lee, M\. Ghassemi, C\. Breazeal, and H\. W\. ParkMdagents: an adaptive collaboration of llms for medical decision\-making\.Advances in Neural Information Processing Systems37,pp\. 79410–79452\.Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p1.1),[2nd item](https://arxiv.org/html/2610.33544#S3.I1.i2.p1.1)\.
- Korngiebel and Mooney \(2021\)D\. M\. Korngiebel and S\. D\. MooneyConsidering the possibilities and pitfalls of generative pre\-trained transformer 3 \(gpt\-3\) in healthcare delivery\.npj Digital Medicine4\(1\),pp\. 93\.Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p2.1)\.
- Lauet al\.\(2018\)J\. J\. Lau, S\. Gayen, A\. Ben Abacha, and D\. Demner\-FushmanA dataset of clinically generated visual questions and answers about radiology images\.Scientific data5\(1\),pp\. 180251\.Cited by:[3rd item](https://arxiv.org/html/2610.33544#A1.I1.i3.p1.1),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.33544#S3.SS1.p1.1)\.
- Liet al\.\(2023\)C\. Li, C\. Wong, S\. Zhang, N\. Usuyama, H\. Liu, J\. Yang, T\. Naumann, H\. Poon, and J\. GaoLlava\-med: training a large language\-and\-vision assistant for biomedicine in one day\.arXiv preprint arXiv:2306\.00890\.Cited by:[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- Liet al\.\(2025\)D\. Li, Z\. Tan, P\. Qian, Y\. Li, K\. Chaudhary, L\. Hu, and J\. ShenSMoA: improving multi\-agent large language models with s parse m ixture\-o f\-a gents\.InPacific\-Asia Conference on Knowledge Discovery and Data Mining,pp\. 54–65\.Cited by:[2nd item](https://arxiv.org/html/2610.33544#S3.I1.i2.p1.1)\.
- Liuet al\.\(2026\)A\. H\. Liu, K\. Khandelwal, S\. Subramanian, V\. Jouault, A\. Rastogi, A\. Sadé, A\. Jeffares, A\. Jiang, A\. Cahill, A\. Gavaudan,et al\.Ministral 3\.arXiv preprint arXiv:2601\.08584\.Cited by:[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- Liuet al\.\(2021\)B\. Liu, L\. Zhan, L\. Xu, L\. Ma, Y\. Yang, and X\. WuSlake: a semantically\-labeled knowledge\-enhanced dataset for medical visual question answering\.In2021 IEEE 18th international symposium on biomedical imaging \(ISBI\),pp\. 1650–1654\.Cited by:[1st item](https://arxiv.org/html/2610.33544#A1.I1.i1.p1.1),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.33544#S3.SS1.p1.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- Mooret al\.\(2023\)M\. Moor, Q\. Huang, S\. Wu, M\. Yasunaga, Y\. Dalmia, J\. Leskovec, C\. Zakka, E\. P\. Reis, and P\. RajpurkarMed\-flamingo: a multimodal medical few\-shot learner\.InMachine learning for health \(ML4H\),pp\. 353–367\.Cited by:[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- OpenAI \(2024\)OpenAIGPT\-4 technical report\.External Links:2303\.08774Cited by:[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2\.2](https://arxiv.org/html/2610.33544#S2.SS2.SSS0.Px1.p1.1)\.
- Tanget al\.\(2024\)X\. Tang, A\. Zou, Z\. Zhang, Z\. Li, Y\. Zhao, X\. Zhang, A\. Cohan, and M\. GersteinMedagents: large language models as collaborators for zero\-shot medical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 599–621\.Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.33544#S2.SS2.p2.2),[2nd item](https://arxiv.org/html/2610.33544#S3.I1.i2.p1.1)\.
- Team \(2024\)G\. TeamGemini: a family of highly capable multimodal models\.External Links:2312\.11805Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1)\.
- Teamet al\.\(2025\)K\. Team, Y\. Bai, Y\. Bao, Y\. Charles, C\. Chen, G\. Chen, H\. Chen, H\. Chen, J\. Chen, N\. Chen,et al\.Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)J\. Wang, W\. Jue, B\. Athiwaratkun, C\. Zhang, and J\. ZouMixture\-of\-agents enhances large language model capabilities\.InThe Thirteenth International Conference on Learning Representations,Cited by:[2nd item](https://arxiv.org/html/2610.33544#S3.I1.i2.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, Y\. Zhu, H\. Zhao, X\. Zheng, D\. Sui, T\. Wang, W\. Tang, Y\. Wang, E\. Harrison, C\. Pan,et al\.Colacare: enhancing electronic health record modeling through large language model\-driven multi\-agent collaboration\.InProceedings of the ACM on Web Conference 2025,pp\. 2250–2261\.Cited by:[2nd item](https://arxiv.org/html/2610.33544#S3.I1.i2.p1.1)\.
- Xiaet al\.\(2026\)P\. Xia, J\. Wang, Y\. Peng, K\. Zeng, Z\. Dong, X\. Wu, X\. Tang, H\. Zhu, Y\. Li, L\. Zhang, S\. Liu, Y\. Lu, and H\. YaoMMedAgent\-rl: optimizing multi\-agent collaboration for multimodal medical reasoning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.33544#S2.SS2.p2.2),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.p1.1)\.
- Zhanget al\.\(2024a\)K\. Zhang, R\. Zhou, E\. Adhikarla, Z\. Yan, Y\. Liu, J\. Yu, Z\. Liu, X\. Chen, B\. D\. Davison, H\. Ren,et al\.A generalist vision–language foundation model for diverse biomedical tasks\.Nature Medicine,pp\. 1–13\.Cited by:[1st item](https://arxiv.org/html/2610.33544#S3.I1.i1.p1.1)\.
- Zhanget al\.\(2024b\)X\. Zhang, C\. Wu, Z\. Zhao, W\. Lin, Y\. Zhang, Y\. Wang, and W\. XieDevelopment of a large\-scale medical visual question\-answering dataset\.Communications Medicine4,pp\. 277\.Cited by:[4th item](https://arxiv.org/html/2610.33544#A1.I1.i4.p1.1),[§2\.3](https://arxiv.org/html/2610.33544#S2.SS3.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.33544#S3.SS1.p1.1)\.
- Zhuet al\.\(2025\)Y\. Zhu, Z\. He, H\. Hu, X\. Zheng, X\. Zhang, Z\. Wang, J\. Gao, L\. Ma, and L\. YuMedAgentBoard: benchmarking multi\-agent collaboration with conventional methods for diverse medical tasks\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2610.33544#S1.p2.1)\.

Appendix

- •§[A](https://arxiv.org/html/2610.33544#A1)Implementation Details - –Training Details - –Hyper\-Parameters - –Datasets - –Evaluation Metrics
- •§[B](https://arxiv.org/html/2610.33544#A2)Derivation of the Reward\-Induced Decision Threshold
- •§[C](https://arxiv.org/html/2610.33544#A3)Additional Analysis - –Routing Validation - –Risk–Coverage Analysis - –False\-Certainty Safety Analysis - –Statistical Significance - –Extra Judge Evaluation
- •§Examples ofSMART\-Med - –Example 1: Pathology Question on Inflammatory Infiltrate Characterization - –Example 2: Multiple\-Choice Question on Muographic Map Interpretation - –Discussion
- •§[D](https://arxiv.org/html/2610.33544#A4)Showcases ofMedVR\-3K
- •§[E](https://arxiv.org/html/2610.33544#A5)Prompts Design - –Prompts for BuildingMedVR\-3K - –Prompts for BuildingSMART\-Med

## Appendix AImplementation Details

#### Training Details\.

We adopt the Qwen3\-VL\-4B\-Instruct\[[1](https://arxiv.org/html/2610.33544#bib.bib23)\]as the foundation model due to the trade\-off between model scale and performance, as well as its superior instruction\-following capability in multimodal settings\. We further post\-train the model on our constructedMedVR\-3Kto enhance its reasoning ability and uncertainty estimation\. Specifically, we employ Low\-Rank Adaptation \(LoRA\)\[[7](https://arxiv.org/html/2610.33544#bib.bib27)\]to fine\-tune all linear layers of the model for both supervised fine\-tuning \(SFT\) and reinforcement learning\. During SFT, we train the model for 5 epochs with a learning rate of1×10−41\\times 10^\{\-4\}\. For the subsequent GRPO stage, we adopt a smaller learning rate of1×10−61\\times 10^\{\-6\}and fine\-tune for 1 epoch to stabilize policy optimization\. We follow a curriculum learning strategy during training\. The model is first trained on samples with lower difficulty levels and progressively exposed to more challenging samples, enabling it to gradually acquire complex reasoning capabilities\. In the GRPO stage, we focus exclusively on high\-difficulty samples to further push the upper bound of the model’s reasoning performance and decision\-making ability under uncertainty\. For the VLM reward model, we employ the Qwen3\-VL\-2B\-Instruct and fine\-tune it on the response\-quality pairs offered byMedVR\-3Kfor 10 epochs\. During GRPO fine\-tuning, it will be frozen and used to output the quality rewards\. All experiments are conducted on 8 NVIDIA Tesla H100 GPUs\. We implement our training pipeline based on the ms\-swift framework\. We set the number of generations for GRPO is 6 and the number of gradient accumulation steps is 3\.

#### Hyper\-Parameters\.

In our experiments, we set the maximum number of specialists to 3 and discussion rounds to 2 for both baselines and ourSMART\-Med\. All agents are initialized from Qwen3\-VL\-4B\-Instruct, except forSMART\-Med\*, which uses a version fine\-tuned on ourMedVR\-3K\. Unless otherwise noted, our implementation follows the official repository\.

Table 4:Dataset statistics\. We compare the modality, size, question type of 4 datasets as well as our constructedMedVR\-3Kthat are used in our work\.SlakePathVQAVQA\-RADPMC\-VQAMedVR\-3KSize4,71726,5161,7784,0003,477Answer✓✓✓✓✓Reasoning✗✗✗✗✓Difficulty✗✗✗✗✓ModalityRadiology imagesPathology slidesRadiology imagesRadiology imagesPathology \+ RadiologyQuestion TypeYES/NO/OPENYES/NO/OPENYES/NO/OPENA/B/C/DYES/NO/A/B/C/D

#### Datasets\.

In this work, we employ 4 popular MedVQA datasets as follows:

- •Slakeis a multilingual medical VQA dataset constructed from radiology images, including X\-ray, CT, and MRI, with annotations provided in both English and Chinese, and emphasizes visual grounding with well\-aligned question–answer pairs\[[14](https://arxiv.org/html/2610.33544#bib.bib25)\]\.
- •PathVQAfocuses on pathology images, containing high\-resolution histopathological slides with questions that require fine\-grained visual reasoning and domain\-specific knowledge\[[6](https://arxiv.org/html/2610.33544#bib.bib9)\]\.
- •VQA\-RADis a radiology VQA benchmark built upon clinical radiology images, where questions often involve abnormality detection, modality recognition, and anatomical reasoning; it is relatively small but widely used for evaluating medical reasoning capability\[[10](https://arxiv.org/html/2610.33544#bib.bib26)\]\.
- •PMC\-VQAis a large\-scale dataset derived from biomedical literature, consisting of hundreds of thousands of image–question–answer pairs spanning multiple modalities \(e\.g\., radiology, microscopy, charts\)\[[26](https://arxiv.org/html/2610.33544#bib.bib18)\]\.

We employ a part of their training set to construct ourMedVR\-3Kand their test set for evaluation, and the detailed statistics of each dataset are summarized in Table[4](https://arxiv.org/html/2610.33544#A1.T4)\. In particular, after quality control, we adopted 746, 721, 707, 1303 samples from Slake, PathVQA, VQA\-RAD, and PMC\-VQA respectively\.

#### Evaluation Metrics\.

We adopt two complementary evaluation metrics to assess model performance under different question formats\. \(i\) Accuracy: For multiple\-choice questions, we use exact\-match accuracy, which measures whether the predicted answery^\\hat\{y\}matches the ground\-truth labely⋆y^\{\\star\}:

Accuracy=1N​∑i=1N𝕀⁡\(y^i=yi⋆\),\\text\{Accuracy\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\(\\hat\{y\}\_\{i\}=y\_\{i\}^\{\\star\}\),\(20\)where𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\. This metric is suitable for structured questions with a predefined set of candidate answers\. \(ii\) Correctness: For free\-form questions, exact matching is often overly restrictive due to linguistic variability\. Therefore, we adopt a semantic correctness metric based on an independent and proprietary MLLM evaluator\. Specifically, given a generated answery^\\hat\{y\}and ground\-truth answery⋆y^\{\\star\}\. In practice, we employ the GPT 5\.4 owing to its strong text information processing ability\. Formally, the evaluator modelℰ\\mathcal\{E\}produces a score:

Correctness=ℰ⁡\(y^,y⋆\)∈\[0,1\],\\text\{Correctness\}=\\mathcal\{E\}\(\\hat\{y\},y^\{\\star\}\)\\in\[0,1\],\(21\)which reflects the semantic alignment between the prediction and the reference\.

## Appendix BDerivation of the Reward\-Induced Decision Threshold

We provide a derivation of the reward\-induced decision threshold discussed in Sec\.[2\.2](https://arxiv.org/html/2610.33544#S2.SS2)\. At the router level, the model chooses between answering directly and declaringUNCERTAIN, which triggers multi\-agent collaboration\. We consider an idealized decision setting in whichp⁡\(x,q\)p\(x,q\)is calibrated and the router selects the action with the highest expected certainty\-aware reward\.

Under the certainty\-aware reward in Eq\. \([5](https://arxiv.org/html/2610.33544#S2.E5)\), the expected utilities of the two actions are

Uans=p⁡\(x,q\)−\(1−p⁡\(x,q\)\)​β,Udef=p⁡\(x,q\)​α\.U\_\{\\text\{ans\}\}=p\(x,q\)\-\(1\-p\(x,q\)\)\\beta,\\qquad U\_\{\\text\{def\}\}=p\(x,q\)\\alpha\.\(22\)
The Bayes\-optimal action under this reward selects the action with higher expected utility:

Uans≥Udef\.U\_\{\\text\{ans\}\}\\geq U\_\{\\text\{def\}\}\.\(23\)
Substituting the above utilities, we obtain

p⁡\(x,q\)−\(1−p⁡\(x,q\)\)​β≥p⁡\(x,q\)​α\.p\(x,q\)\-\(1\-p\(x,q\)\)\\beta\\geq p\(x,q\)\\alpha\.\(24\)
Rearranging terms gives

p⁡\(x,q\)​\(1\+β−α\)≥β,p\(x,q\)\(1\+\\beta\-\\alpha\)\\geq\\beta,\(25\)which yields the reward\-induced threshold

p⁡\(x,q\)≥β1−α\+β\.p\(x,q\)\\geq\\frac\{\\beta\}\{1\-\\alpha\+\\beta\}\.\(26\)
Therefore, the corresponding Bayes\-optimal decision rule under the idealized calibrated setting is

aBayes∗​\(x,q\)=\{ANS,p⁡\(x,q\)≥τBayes,DEF,p⁡\(x,q\)<τBayes,τBayes=β1−α\+β\.a\_\{\\mathrm\{Bayes\}\}^\{\*\}\(x,q\)=\\begin\{cases\}\\mathrm\{ANS\},&p\(x,q\)\\geq\\tau\_\{\\mathrm\{Bayes\}\},\\\\ \\mathrm\{DEF\},&p\(x,q\)<\\tau\_\{\\mathrm\{Bayes\}\},\\end\{cases\}\\qquad\\tau\_\{\\mathrm\{Bayes\}\}=\\frac\{\\beta\}\{1\-\\alpha\+\\beta\}\.\(27\)
This threshold should be interpreted as a theoretical utility indifference point induced by the reward parametersα\\alphaandβ\\beta, rather than as the exact decision boundary of the GRPO\-trained router\. In the actual model, GRPO learns a policyπθ⋆​\(a∣x,q\)\\pi\_\{\\theta^\{\\star\}\}\(a\\mid x,q\), whose routing boundary is implicitly determined by

πθ⋆​\(ANS∣x,q\)=πθ⋆​\(DEF∣x,q\)\.\\pi\_\{\\theta^\{\\star\}\}\(\\mathrm\{ANS\}\\mid x,q\)=\\pi\_\{\\theta^\{\\star\}\}\(\\mathrm\{DEF\}\\mid x,q\)\.\(28\)Consequently, the empirical routing boundary may differ fromτBayes\\tau\_\{\\mathrm\{Bayes\}\}due to imperfect calibration, finite training data, optimization dynamics, and the parameterization of the learned policy\.

## Appendix CAdditional Analysis

#### Routing Validation\.

To verify that the learned router agent can exploit meaningful sample\-level uncertainty rather than merely changing the average collaboration rate, we compareSMART\-Med\* against two extreme policies \(always\-single, always\-collaborate\) using the same fine\-tuned backbone and a random\-routing control that replaces the learned router with an independent Bernoulli decision at a matched collaboration rate\. Table\(a\) showsSMART\-Med\* achieves the best accuracy on both MC and FF while issuing only 1\.2 calls on average, outperforming both extremes; Table\(b\) shows random routing achieves far lower accuracy than learned routing despite a comparable single\-agent rate, indicating the router identifies*which*queries benefit from collaboration rather than just how often to invoke it\.

Table 5:Risk–coverage analysis before/after certainty\-aware GRPO\.ModelAURC↓\\downarrowAcc\.@100%↑\\uparrowSMART\-Med0\.266966\.0%SMART\-Med\*0\.233471\.5%
#### Risk–Coverage Analysis\.

We further quantify calibration quality via the Area Under the Risk–Coverage Curve \(AURC; lower is better\)\. Table[5](https://arxiv.org/html/2610.33544#A3.T5)shows certainty\-aware GRPO reduces overall AURC from 0\.2669 to 0\.2334 while improving full\-coverage accuracy from 66\.0% to 71\.5%, confirming that our designed GRPO fine\-tuning improves calibration and accuracy jointly\.

#### False\-Certainty Safety Analysis\.

The false\-certainty is a typical failure mode in MedVQA\. Tablecross\-tabulates the router’s certainty token against correctness before and after GRPO: the false\-certainty rate \(Incorrect\+CERTAIN\) drops from 28\.8% to 8\.6%, while incorrect predictions correctly flaggedUNCERTAINrise from 5\.3% to 15\.7%, showing that certainty\-aware GRPO specifically suppresses overconfident errors and defers the remaining difficult cases to collaboration, which can then correct them \(concrete examples can be found in the Appendix\)\.

#### Statistical Significance\.

We perform paired significance tests using per\-question performance ofSMART\-Med\* against each baseline across the five runs, reported in Table\(a\)\. The improvements over Reconcile \(p=0\.0025p=0\.0025\) and ColaCare \(p=0\.0043p=0\.0043\) are significant at the 1% level, and the improvements over MedAgent \(p=0\.0137p=0\.0137\) and MDAgent \(p=0\.0341p=0\.0341\) are significant at the 5% level\. The comparison with MOA does not reach significance \(p=0\.0860p=0\.0860\), which is consistent with the relatively small performance gap between the two methods observed in Table[1](https://arxiv.org/html/2610.33544#S3.T1)\. Overall, the pairwise comparisons are statistically significant at different salient levels, supporting that the accuracy gains ofSMART\-Med\* are statistically salient against most of baseline MAS models\.

#### Extra Judge Evaluation\.

To verify the reasoning quality improvements of our reasoning quality reward across different judge models, we additionally score the same 100 sampled responses with Gemini 3 Pro, a model that is completely independent of bothMedVR\-3Kconstruction and reward\-model training, and report the overall average score \(across the Logic Consistency, Factual Correctness, and Image Alignment dimensions used in Table[2](https://arxiv.org/html/2610.33544#S3.T2)\) under both judges in Table\(b\)\. The ranking is largely preserved across the two judges, andSMART\-Med\* remains the top\-scoring method under both GPT\-5\.4 \(3\.50\) and Gemini 3 Pro \(3\.73\)\. Notably, Gemini 3 Pro assignsSMART\-Med\* an even higher score than GPT\-5\.4 does, and the relative ordering of the multi\-agent baselines \(e\.g\., ColaCare and MOA scoring higher under Gemini 3 Pro than under GPT\-5\.4\) shifts somewhat but does not change the overall conclusion\. This cross\-judge consistency indicates that the reasoning\-quality advantage ofSMART\-Med\* reflects a genuine property of the generated reasoning traces rather than a preference specific to GPT\-5\.4\.

![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/examples/synpic28718.jpg)\(a\)L1
![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/examples/train_0422.jpg)\(b\)L2 and L6
![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/examples/train_0473.jpg)\(c\)L3
![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/examples/PMC8420045_F3.jpg)\(d\)L4
![Refer to caption](https://arxiv.org/html/2610.33544v1/imgs/examples/PMC8442220_f2.jpg)\(e\)L5

Figure 5:The medical images used in the Listing[1](https://arxiv.org/html/2610.33544#LST1)to Listing[6](https://arxiv.org/html/2610.33544#LST6)\. Notably, the image \(b\) is used for two queries, a multi\-choice question \(L2\) and a free\-form question \(L6\)\.

## Appendix DShowcases ofMedVR\-3K

We present and analyze samples fromMedVR\-3Kto showcase the diversity, reasoning quality, and the reliability of multi\-dimensional annotations\. The corresponding images are shown in Fig\.[5](https://arxiv.org/html/2610.33544#A3.F5)\.

The selected cases span multiple modalities and knowledge domains, including histopathology \(Listings[2](https://arxiv.org/html/2610.33544#LST2),[3](https://arxiv.org/html/2610.33544#LST3), and[6](https://arxiv.org/html/2610.33544#LST6)\), radiology \(Listings[1](https://arxiv.org/html/2610.33544#LST1)and[5](https://arxiv.org/html/2610.33544#LST5)\), and structural biology \(Listing[4](https://arxiv.org/html/2610.33544#LST4)\), demonstrating the broad coverage ofMedVR\-3K\. Importantly, the dataset captures a wide spectrum of reasoning behaviors\. For instance, Listing[2](https://arxiv.org/html/2610.33544#LST2)shows a case where the model reaches the correct conclusion \(CK7 staining of biliary structures\) but introduces a domain\-specific factual error, reflecting realistic partial understanding\. Listing[1](https://arxiv.org/html/2610.33544#LST1)illustrates a low\-quality response with minimal reasoning and weak image grounding despite a correct final answer, while Listing[3](https://arxiv.org/html/2610.33544#LST3)provides a strong positive example with accurate domain knowledge and precise visual alignment in identifying collagen under trichrome staining\. Listing[4](https://arxiv.org/html/2610.33544#LST4)further highlights cases where answers are correct but supported by weak or generic reasoning, whereas Listing[5](https://arxiv.org/html/2610.33544#LST5)demonstrates high\-quality spatial reasoning and robust image grounding in a radiological context\. Finally, Listing[6](https://arxiv.org/html/2610.33544#LST6)captures a subtle semantic discrepancy, where the model’s description \(bile duct and ductules\) is close but not strictly equivalent to the ground\-truth concept \(bile duct cells and canals of Hering\), reflecting the fine\-grained nature of medical terminology\.

Meanwhile, for Listing[5](https://arxiv.org/html/2610.33544#LST5), we also compare it with the doctor evaluation \(i\.e\., Listing[7](https://arxiv.org/html/2610.33544#LST7)\)\. Both responses reach the exact same conclusion and share the same key reasoning: the gas is located outside the necrotic right kidney within the retroperitoneal space, while the gallbladder, left kidney, and stomach are all anatomically incorrect locations\. The main differences lie in style and structure: the physician’s response is more detailed and pedagogical, presenting a clear step\-by\-step elimination of each distractor without needing to cross\-reference Fig\.[5](https://arxiv.org/html/2610.33544#A3.F5)\(e\)\-B\. Although the MLLM’s response is more direct in reaching the answer, it still contains the key analytical reasoning chain \(e\.g\., step\-by\-step elimination\), demonstrating that our constructed dataset is well\-equipped with human\-doctor\-level reasoning chains\.

For the associated quality annotations inMedVR\-3K, including logical consistency, factual correctness, and image alignment—closely match expert judgment across these varied scenarios\. They successfully distinguish between correct answers with flawed reasoning \(Listings[2](https://arxiv.org/html/2610.33544#LST2)and[4](https://arxiv.org/html/2610.33544#LST4)\), unsupported predictions \(Listing[1](https://arxiv.org/html/2610.33544#LST1)\), high\-quality reasoning \(Listings[3](https://arxiv.org/html/2610.33544#LST3)and[5](https://arxiv.org/html/2610.33544#LST5)\), and semantically imprecise outputs \(Listing[6](https://arxiv.org/html/2610.33544#LST6)\)\. This demonstrates thatMedVR\-3Knot only provides diverse and clinically relevant VQA samples, but also enables nuanced and reliable evaluation of model behavior\. Overall,MedVR\-3Kserves as a challenging and well\-calibrated benchmark for advancing medical multimodal reasoning\.

Listing 1:Example 11\{

2"question":"Areairfluidlevelsdepictedinthisimage?",

3"image\-path":"synpic28718\.jpg",

4"response":"I’mnot100%sureformyanalysistothismedicalimageandthegivenquestion\.But,IcanguesstheanswerisYES\.\\n\\n<answer\>YES</answer\>",

5"label":"YES",

6"logic\_consistency":1,

7"factual\_correctness":3,

8"image\_alignment":1,

9"difficulty":2,

10"judgment":"Theassistantprovidesessentiallynoreasoningandexplicitlystatesuncertainty,thengivesabareYESanswer,sotheconclusionisunsupportedandlogicallyweak\.Factualcorrectnessisdifficulttoverifyfromtheresponsealone;theimagedoesappeartoshowintraluminalgas\-fluidinterfaces,sothefinalanswermaybecorrect,butthelackofexplanationandhedgingpreventahigherscore\.ImagealignmentispoorbecausetheassistantdoesnotreferenceanyspecificvisualfeaturesorregionsintheCTimage\.Thetaskitselfisabasicimage\-identificationquestionrequiringlimitedmedicalknowledge,sodifficultyislow\."

11\}

Listing 2:Example 21\{

2"question":"ArebileductcellsandcanalsofHeringstainedherewithanimmunohistochemicalstainforcytokeratin7?",

3"image\-path":"train\_0422\.jpg",

4"response":"Theimagedisplaysahistologicalsectionwithcellsstainedtorevealspecificstructures\.Thepresenceofdarklystained,duct\-likestructures\(indicatedbyarrows\)amidstabackgroundofsmaller,round,blue\-stainednucleiisconsistentwiththeappearanceofbileductsandcanalsofHeringunderimmunohistochemicalstaining\.Cytokeratin7\(CK7\)isawell\-knownmarkerthatstainsbileductepitheliumandcanalsofHering,whicharetypicallypositiveforCK7\.Thestainingpatternintheimage,withstrong,specificlabelingoftheductalstructures,alignswiththeexpectedimmunohistochemicalprofileforCK7\.Thesurroundingcellsappeartobehepatocytes,whicharealsoknowntoexpressCK7,butthemostprominentstainingisintheductalstructures\.Giventhisevidence,thestainingisconsistentwithanimmunohistochemicalstainforcytokeratin7\.<answer\>YES</answer\>"

5"label":"YES",

6"logic\_consistency":3,

7"factual\_correctness":2,

8"image\_alignment":4,

9"difficulty":2,

10"judgment":"Theresponsegivesacoherentrationalelinkingbrownimmunostainingofduct\-likestructurestoCK7\-positivebileducts/canalsofHering,sothelogicisgenerallyunderstandablebutsomewhatoverstatedandnotfullyrigorous\.Imagegroundingisreasonablygoodbecauseitreferstothedarkbrownductalstructuresindicatedbyarrowsagainstbluenuclei,whichmatchesthevisualfeatures\.However,factualaccuracyisreducedbytheclaimthatsurroundinghepatocytesarealsoknowntoexpressCK7;inroutineliverhistology,hepatocytesaregenerallyCK7\-negative,whilebileductepitheliumandcanalsofHeringareCK7\-positive\.Thatisasignificantpathology\-specificerror,thoughthefinalYESanswerislikelycorrect\.Thequestionitselfisabasichistopathology/IHCidentificationtaskrequiringlimitedspecialistknowledge,sodifficultyislow\-to\-moderate\."

11\}

Listing 3:Example 31\{

2"question":"IsCollagenstainedbluebythetrichromestain?",

3"image\-path":"train\_0473\.png",

4"response":"Inhistologicalstaining,thetrichromestainiscommonlyusedtodifferentiatecollagenfromothertissuecomponents\.Inthisstain,collagenfibersaretypicallystainedblueorgreen,whilemusclefibersappearredorpink,andnucleiareoftencounterstainedwithadifferentcolor\(e\.g\.,purpleorblack\)\.\\n\\nIntheprovidedimage,thebackgroundmatrixappearspredominantlyblue,andtherearescatteredpinkish\-purplestructuresthatresemblenucleiorothercellularcomponents\.Theblue\-stainedareashaveafibrous,wavyappearanceconsistentwithcollagenfibers\.\\n\\nTherefore,basedonthevisualevidenceandstandardhistologicalknowledge,collagenisstainedbluebythetrichromestain\.\\n\\n<answer\>YES</answer\>",

5"label":"YES",

6"logic\_consistency":5,

7"factual\_correctness":5,

8"image\_alignment":5,

9"difficulty":3,

10"judgment":"Theassistantprovidesaclear,step\-by\-stepexplanationlinkingtrichromestainingprinciplestotheimagefeatures,correctlyidentifyingblue\-stainedfibrousareasascollagen\.Thedescriptionofothercomponents\(pinknuclei,background\)alignswithstandardhistology\.Theresponseisfactuallyaccurateanddemonstratesprecisevisualanalysisoftheimage,thoughthequestionitselfismoderateincomplexity,requiringbasichistologicalknowledgeandsingle\-stepreasoning\."

11\}

Listing 4:Example 41\{

2"question":"WhatdothecoloredspheresinFigureBrepresent?A:Ligandpockets\.B:Beta\-barrels\.C:Type\-Iresidues\.D:SDPresidues\.",

3"image\-path":"PMC8420045\_F3\.jpg",

4"response":"IcandiscernvariouscoloredspheresacrosstheproteinstructuresshowninFigureB\.Thesespheresseemtohighlightspecifictypesofresiduesbasedontheirchemicalcharacteristics\.\\n\\nFromtheimage:\\n\-Thespheresareshownindifferentcolors\(suchasred,blue,andwhite\),whichcouldindicatespecificfeaturesoftheresidues\.\\n\-Thecontextofthesurroundingfigures\(whichdisplayproteinsandtheirstructuralfeatures\)suggeststhatthecoloredspheresarelikelyhighlightingspecificresiduesthatarecrucialforparticularinteractionsorfunctions\.\\n\\nGiventheevidenceintheimageandtheprovidedoptions,themostplausibleconclusionisthatthesespheresrepresent\*\*Type\-Iresidues\*\*,astheyaretypicallyassociatedwithspecificaminoacidpositionscriticalforstructuralfunctionsinproteinanalysis\.\\n\\n<think\>Basedonthevisualizationandthecontextofproteinanalysis,thecoloredspheresinFigureBmostlikelyrepresentType\-Iresidues\.</think\><answer\>C</answer\>",

5"label":"C",

6"logic\_consistency":5,

7"factual\_correctness":5,

8"image\_alignment":4,

9"difficulty":3,

10"judgment":"TheresponsecorrectlyidentifiestheanswerasC\(Type\-Iresidues\),matchingthelabel,sofactualcorrectnessisfull\.Theinternalreasoningiscoherentandnon\-contradictory,thoughitlacksstrongevidencefromthefigure;logicconsistencyisstillhigh\.Withoutseeingtheimage,alignmentisjudgedasreasonable\(4\)becausethedescriptionofcoloredspheresisplausibleforaproteinfigure,andthecorrectchoicesuggestsgoodvisualmatching\.Difficultyismoderateasthequestionrequiresknowledgeofproteinstructuralfeaturesbutisnotextremelyspecialized\."

11\}

Listing 5:Example 51\{

2"question":"WhatisthebottomasteriskpointingtointheimageA?A:IntramuralgasinthegallbladderB:GasextendingthroughtheretroperitoneumC:ExtensiveparenchymaldestructionoftheleftkidneyD:Inflammationinthestomach\.",

3"image\-path":"PMC8442220\_f2\.jpg",

4"response":"<think\>ThecoronalCTimageAshowsthepatient’srightsideontheleftoftheimage\.Thereisamarkedlyabnormalrightrenalareawithextensivegasreplacingmuchoftherenalparenchyma\.Theupperasteriskiswithinthisgas\-filleddestructiverenalregion\.Thebottomasteriskpointstostreaky/mottledgastrackingoutsidethekidneyalongtherightflank/retroperitonealspace\.Thisisnotthegallbladder\(seeninimageBwitharrow\),nottheleftkidney,andnotthestomach\.</think\><answer\>B</answer\>",

5"label":"B",

6"logic\_consistency":4,

7"factual\_correctness":4,

8"image\_alignment":5,

9"difficulty":3,

10"judgment":"Theassistantgivesaclearrationale:itdistinguishesthelowerasteriskfromtheupperrenalgascollection,notesthatthelowermarkerindicatesgastrackingoutsidethekidneyalongtheflank/retroperitoneum,andexplicitlyrulesoutthegallbladder,leftkidney,andstomach\.ThisislogicallycoherentandmatchesthevisualappearanceofimageA\.TheanswerchoiceBismedicallyappropriate;theimageshowsemphysematouschangewithgasextendingintotheretroperitoneal/perinephricregionratherthangallbladdergas,leftrenaldestruction,orgastricinflammation\.Imagegroundingisstrongbecausetheresponselocalizesthefindingtothespecificlowerasteriskandrelatesittosurroundinganatomy\.Aperfectlogic/factualscoreisnotwarrantedbecausethereasoningisconciseratherthanexhaustive,anditreferencesimageBtoexcludethegallbladdereventhoughthequestionasksonlyaboutimageA\.Thequestionitselfisofmoderatedifficulty,requiringstandardradiologicanatomyandpathologyrecognitionratherthanhighlyadvancedinference\."

11\}

Listing 6:Example 61\{

2"question":"Whatarestainedherewithanimmunohistochemicalstainforcytokeratin7?",

3"image\-path":"train\_0422\.jpg",

4"response":"<think\>Theimageisamicrographoflivertissuewhereanimmunohistochemicalstain\(brown\)hasbeenapplied,likelyforcytokeratin7\.Thethinarrowpointstoadistinct,lumen\-containingstructurewithinaportaltract,whichismorphologicallycharacteristicofabileduct\.Thethickarrowpointstosmaller,irregularclustersofstainedcellsattheperipheryoftheportaltract,representingbileductules\.Bothofthesebiliaryepithelialstructuresareknowntobepositiveforcytokeratin7,whilethesurroundinghepatocytesarenegative\.</think\><answer\>Abileduct\(thinarrow\)andbileductules\(thickarrow\)</answer\>",

5"label":"BILEDUCTCELLSANDCANALSOFHERING",

6"logic\_consistency":5,

7"factual\_correctness":3,

8"image\_alignment":3,

9"difficulty":3,

10"judgment":"Theresponseisinternallylogicalandclearlydescribesthetwoarrowlabeledstructures\.However,thelabelspecifies’bileductcellsandcanalsofHeringm’whiletheanswersays’bileductandbileductules’,bileductulesarenotexactlythesameascanalsofHering\(thelatteraretransitionalchannelsbetweencanaliculiandductules\),sofactualcorrectnessisslightlyreduced\.Withoutseeingtheactualimage,thealignmentscoreisamoderate3,assumingthearrowsmightnotperfectlymatchthelabeledstructures\.Difficultyisaverage\(3\)becauseCK7staininginliverisastandardhistopathologyconcept\."

11\}

Listing 7:Human Doctor Evluation to Example 51\{

2"question":"WhatisthebottomasteriskpointingtointheimageA?A:IntramuralgasinthegallbladderB:GasextendingthroughtheretroperitoneumC:ExtensiveparenchymaldestructionoftheleftkidneyD:Inflammationinthestomach\.",

3"image\-path":"PMC8442220\_f2\.jpg",

4"doctor":"\(1\)ImageLocalization:FigureAisacoronalCTimage,withtheleftsideoftheimagecorrespondingtothepatient’srightside\.Alargeareaofair\-containinglow\-attenuationshadowcanbeseenintherightrenalregion,indicatingrenalparenchymalnecrosiswithgasaccumulation\.Theupperasteriskmarkstheintrarenalgasandnecrosis,whilethelowerasteriskmarkstheextrarenalgasshadowlocatedintheretroperitonealspace\.

5

6\(2\)Systematiceliminationofdistractors\!

7A\(Intramuralgasinthegallbladderwall\):Thegallbladderislocatedintherightupperquadrantliverbedarea,whichdoesnotmatchtheanatomicallocationofthelowerasterisk\.ThearrowinFigureBmerelyservesasareferenceforthegallbladderandcanbeexcludedwithoutcross\-referencingbetweenimages\.

8C\(Extensiveparenchymaldestructionoftheleftkidney\):Theleftkidneyislocatedontherightsideoftheimage,withessentiallynormalmorphologyanddensity,showingnogasaccumulationorextensivedestruction\.Thelesionisentirelyontherightsideandcanthereforebeexcluded\.

9D\(Gastricinflammation\):Themainbodyofthestomachisintheleftupperquadrant,posteriortotheliver;thegastricantrumisneartherightupperquadrant,adjacenttothepancreatichead\.Thisdoesnotcorrespondtotheright\-sidedretroperitoneallocationofthelowerasteriskandcanbeexcluded\.

10

11\(3\)Conclusion:Thegasindicatedbythelowerasteriskextendsoutwardfromthenecroticrightkidneyintotheretroperitonealspace,correspondingtooptionB\.",

12\}

## Appendix EPrompts Design

#### Prompts for BuildingMedVR\-3K\.

MedVR\-3Kis a reasoning\-centric dataset, and the reasoning quality is critical\. To ensure the quality, we employ an independent judge MLLM \(i\.e\., GPT 5\.4\) to evaluate the quality of each reasoning chain\. We design the prompts for the judge model as below:

Prompts for the judge MLLMYou are an expert evaluator for multimodal medical QA tasks\. Your role is to critically assess the assistant’s response to the provided User Input\.\#\#\# User’s Question\{question\}\#\#\# Assistant Response\{assistant\_response\}\-\-\-\#\# Evaluation RubricsEvaluate the assistant’s response on the following four dimensions\. Each dimension is scored on a 1\-5 scale\.\#\#\# 1\. Logic Consistency \(1\-5\)Assesses the coherence, structure, and validity of the reasoning chain\.\| Score \| Criteria \|\|\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|\| 1 \| Severe logical fallacies, self\-contradiction, circular reasoning, or complete lack of reasoning \|\| 2 \| Flawed reasoning with significant gaps; conclusion does not logically follow from premises \|\| 3 \| Generally sound reasoning but lacks depth, contains minor leaps, or oversimplifies complexity \|\| 4 \| Clear, structured reasoning that follows medical diagnostic/analytical standards \|\| 5 \| Flawless, step\-by\-step deductive reasoning; robust, complete, and clinically sound \|\#\#\# 2\. Factual Correctness \(1\-5\)Assesses the medical accuracy of the response\.\| Score \| Criteria \|\|\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|\| 1 \| Critical hallucinations; dangerous or completely incorrect medical advice/claims \|\| 2 \| Significant factual errors regarding symptoms, anatomy, pathology, or treatment \|\| 3 \| Mostly correct but contains minor inaccuracies, outdated information, or imprecise terminology \|\| 4 \| Factually correct throughout; no errors detected \|\| 5 \| Factually correct and precise; uses professional terminology accurately and appropriately \|\#\#\# 3\. Image Alignment \(1\-5\)Assesses how well the response utilizes and accurately interprets visual information\.\| Score \| Criteria \|\|\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|\| 1 \| Completely ignores the image; describes features not present or contradicts visual evidence \|\| 2 \| Mentions the image but misinterprets key visual features \(e\.g\., wrong location, wrong modality, incorrect identification\) \|\| 3 \| Generic response that could apply to any similar image; lacks specific visual grounding \|\| 4 \| Correctly identifies and describes relevant visual features; demonstrates basic visual grounding \|\| 5 \| Deep visual grounding; precisely correlates specific image regions with clinical reasoning; demonstrates expert visual analysis \|\#\#\# 4\. Difficulty \(1\-5\)Assesses the complexity of the question itself \(not the answer quality\)\.\| Score \| Criteria \|\|\-\-\-\-\-\-\-\|\-\-\-\-\-\-\-\-\-\-\|\| 1 \| Simple fact retrieval or common knowledge; no medical expertise required \|\| 2 \| Basic identification task; requires minimal medical knowledge \|\| 3 \| Moderate complexity; requires standard medical knowledge and single\-step reasoning \|\| 4 \| Complex case; requires multi\-step reasoning, integration of multiple concepts, or specialist knowledge \|\| 5 \| Highly complex; requires integrating multiple modalities, rare/specialized knowledge, or advanced differential diagnosis \|\-\-\-\#\# Strict Evaluation Guidelines1\. \*\*Be conservative\*\*: A score of 5 is reserved for "perfect" or "exemplary" responses\. Use 4 for "good but not perfect\."2\. \*\*Penalize hallucinations heavily\*\*: Any fabricated information should result in a Factual Correctness score smaller than 2\.3\. \*\*Guess vs\. reasoning\*\*: If the reasoning is missing, flawed, or insufficient, but the final answer is correct \(guessed\), cap the Logic Consistency score at 2\.4\. \*\*Image requirement\*\*: A response that fails to meaningfully engage with the image \(when visual input is essential\) cannot receive an Image Alignment score above 3\.5\. \*\*Response completeness\*\*: If the response is incomplete or refuses to answer without justification, scores should reflect this deficiency\.6\. \*\*Evidence\-based judgment\*\*: All scores must be justified with specific evidence from the assistant’s response\.\-\-\-\#\# Output FormatOutput ONLY a valid JSON object\. Do not include markdown formatting, code blocks, or explanations outside the JSON\.\{\{"scores": \{\{"logic\_consistency": <int 1\-5\>,"factual\_correctness": <int 1\-5\>,"image\_alignment": <int 1\-5\>,"difficulty": <int 1\-5\>\}\},"critique": "<brief, evidence\-based justification for your scores\>"\}\}

#### Prompts for BuildingSMART\-Med\.

In our method, there are three kinds of roles, undertaking different tasks: \(1\) Router: the agent for initial assessment, certainty check and expert recruitment; \(2\) Specialist: the agent is assigned with specific tasks and collaboration patterns in the multi\-agent team; \(3\) Moderator: the agent aims to summarize the analysis reports from multiple agents\. Their system prompts, role prompts and task prompts are designed as follows:

Prompts for the router agent\# System PromptYou are a medical resource allocation router\. Your task is to allocate the medical question to the most appropriate specialist agents efficiently and effectively\.\#\# TaskPlease recruit medical experts for this question: <image\> \{question\}\#\# Instruction1\. Recruit maximally 5 specialist roles relevant to answer this question\. Reference roles: \[Radiologist, Pathologist, Surgeon, Oncologist, Endocrinologist, Geneticist, Cardiologist, Neurologist\]2\. Generate a brief task description for each role explaining their job in the collaboration to correctly answer this question\.3\. For role descriptions, using \*\*\|\*\* to seperate different roles\.\#\# Output Format<role\>\[Role1, Role2, \.\.\.\]</role\>\\n<role\_desc\>\[Role1: Description \| Role2: Description \| \.\.\.\]</role\_desc\>\#\# Example<role\>Radiologist, Oncologist</role\>\\n<role\_desc\>Interprets CT images to detect lung masses \| Evaluates malignancy and treatment strategy</role\_desc\>

Prompts for the specialist agent\# System PromptYour task is \{role\_description\}, and your job is \{task\_description\}\.\#\# TaskAnalyze the provided medical image and answer the following \{answer\_type\} question: <image\>\{question\}\#\# Instructions1\. \*\*Visual Analysis\*\*: Thoroughly examine the image and identify all anatomical structures, abnormalities, or features relevant to the question\.2\. \*\*Evidence\-Based Reasoning\*\*: Base your conclusion strictly on the visual evidence\. Cite specific observations that support your analysis\.3\. \*\*Uncertainty Annotation\*\*: If the evidence is genuinely ambiguous or critical features are obscured, return UNCERTAIN in your response\.\#\# Output Format<reason\>\[Your visual analysis citing specific evidence\]</reason\><certainty\>\[CERTAIN/UNCERTAIN\]</certainty\><answer\>\[Your definitive best\-effort answer: \{answer\_type\}\]</answer\>·\#\# Examples\*\*High Confidence Case:\*\*<reason\>The CT slice clearly shows a hyperdense, well\-circumscribed lesion in the right lung base with characteristics typical of a hamartoma\.</reason\><certainty\>CERTAIN</certainty\><answer\>Benign</answer\>\*\*Low Confidence Case:\*\*<reason\>The image shows a possible mass in the mediastinum, but the slice level appears to be above the cardiac silhouette and the structure boundaries are poorly defined due to motion artifact\.</reason\><certainty\>UNCERTAIN</certainty\><answer\>Malignant</answer\>\#\# Critical Rules1\. The \`<certainty\>\` tag MUST be either \`CERTAIN\` or \`UNCERTAIN\` \(no other values allowed\)\.2\. The \`<answer\>\` tag MUST always contain a specific \{answer\_type\} value, regardless of certainty level\.

Prompts for the moderator agent\# System PromptYou are a medical assistant who excels at summarizing and synthesizing based on multiple experts from various domain experts\.\#\# Context\*\*Image\*\*: <image\>\*\*Question\*\*: \{question\}\*\*Answer Type\*\*: \{answer\_type\} \(Your final answer MUST strictly conform to this type\)\*\*Initial Assessment\*\*:\{initial\_assessment\}\*\*Expert Reports\*\*:\{report\}\#\# Analysis Framework\#\#\# 1\. Individual AssessmentFor the given initial assessment and expert reports, evaluate:\- \*\*Certainty Level\*\*: CERTAIN or UNCERTAIN \(from <certainty\> tag\)\- \*\*Key Evidence\*\*: Specific visual features cited from the image\- \*\*Reasoning Quality\*\*: Logical soundness of the inference chain\- \*\*Risk of Hallucination\*\*: Whether claims exceed visible evidence\#\#\# 2\. Cross\-Expert Comparison\- \*\*Consensus Detection\*\*: Identify agreements and disagreements across experts and the initial assessment\- \*\*Evidence Strength\*\*: Which conclusion has stronger visual support?\- \*\*Certainty Weighting\*\*: Prioritize CERTAIN answers unless their evidence is weaker\- \*\*Conflict Analysis\*\*: If experts disagree, determine if the disagreement stems from:\- Different interpretation of the same evidence → weigh reasoning quality\- Different evidence focus → check which evidence is more relevant to the question\- Fundamental ambiguity in the image → acknowledge irreducible uncertainty\#\#\# 3\. Adjudication Rules \(Apply in Order\)1\. \*\*Strong Consensus\*\*: If there are more than two experts certainly agree with strong evidence, adopt their answer2\. \*\*Quality Override\*\*: If UNCERTAIN experts have stronger evidence than CERTAIN ones → favor evidence over certainty label\#\#\# 4\. Meta\-Evaluation \(Critical Step\)Even if experts agree with the initial assessment, ask yourself:\- Is there a \*\*plausible alternative interpretation\*\* they missed?\- Does their conclusion \*\*overreach the visual evidence\*\*?\- Would a \*\*different expert specialty\*\* \(e\.g\., radiologist vs\. pathologist\) disagree?Finally, your task is to determine the answer of this question according to your analysis to the initial assessment and expert reports\.\#\# Output Requirements\*\*CRITICAL\*\*: Your answer MUST be a valid \{answer\_type\} value\.\#\# Output Format<reason\>\[Step\-by\-step adjudication: summarize expert positions, compare evidence quality, explain your reasoning for selecting the final answer\]</reason\><answer\>\[Your final answer\]</answer\>

## Appendix FLimitation and Social Impact

#### Limitation\.

InSMART\-Med, we employ local small\-scale MLLMs as the agent\. Although it can preserve the privacy, save the computational cost and latency, the overall performance is still bounded by the capacity of the underlying backbone model, which may limit its ability to handle extremely complex or rare medical cases compared to very large proprietary models\. Therefore, how to deal with extremely complex or rare medical cases can be a potential future direction\.

#### Social Impact\.

This work has important implications for the development of privacy\-preserving and deployable medical AI systems\. By enabling fully local inference and reducing reliance on cloud\-based models,SMART\-Medhelps mitigate data privacy and regulatory concerns in clinical environments\. Moreover, the integration of uncertainty\-aware decision\-making can improve the reliability and safety of AI\-assisted diagnosis by reducing overconfident errors\. However, potential risks remain: incorrect or biased reasoning generated by the model could still mislead practitioners if used without proper oversight, and the dataset construction process may inherit biases from source data and teacher models\. Therefore,SMART\-Medshould be deployed as a decision\-support tool rather than a replacement for medical professionals, and future work should focus on improving robustness, fairness, and clinical validation\.