Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agents

# Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agents
Source: [https://arxiv.org/html/2605.26691](https://arxiv.org/html/2605.26691)
Yunhui Gan1,2,3, Tan Pan1,2,511footnotemark:1, Kaiyu Guo4,2, Limei Han1,2 Weimiao Yu5, Guangnan Ye1,3, Chen Jiang1,222footnotemark:2, Yuan Cheng1,2,322footnotemark:2 1Fudan University2Shanghai Academy of Artificial Intelligence for Science 3Shanghai Innovation Institute4The University of Queensland 5Bioinformatics Institute \(BII\), Agency for Science, Technology and Research \(A\*STAR\) \{yhgan23,pant23\}@m\.fudan\.edu\.cn kaiyu\.guo@uq\.edu\.au, jiangchen@sais\.org\.cn, cheng\_yuan@fudan\.edu\.cn

###### Abstract

Medical AI agents increasingly use external tools for diagnosis, treatment recommendation, and evidence retrieval, yet most existing approaches assume that task\-appropriate tools are reliable within their intended scope\. This assumption is fragile in real clinical settings, where even relevant tools may fail on challenging instances and lead to unsafe downstream decisions\. To address this issue, we study medical tool use under imperfect\-tool settings to correct failure instances missed by individual tools\. Instance\-dependent failure patterns create a gap between the best fixed single tool and an ideal instance\-wise selector, which we refer to as the Single\-Oracle risk gap\. The core challenge is that conventional task\-level tool selection cannot realize this gap, as it is inherently bounded by the performance of the best single tool\. Motivated by this observation, we therefore account for instance\-level heterogeneity and formulate tool use as an instance\-level selection problem\. Particularly, we propose a GRPO\-based reinforcement learning framework with rewards for probabilistic risk minimization and disagreement\-aware synergy learning, which promotes instance\-level correction of erroneous tool consensus\. Furthermore, an entropy\-guided sampling strategy is adopted to upweight high\-disagreement instances, which provide stronger signals for learning instance\-specific tool synergy\. These two components complement each other in mitigating instance\-level heterogeneity and improving tool synergy\. Experiments on two tasks and seven medical benchmarks show that our method consistently achieves robust and stable improvements over a broad range of baselines, highlighting the importance of synergy\-aware tool use for reliable medical agentic systems\.

## 1Introduction

Recent advances in agentic systems have increasingly emphasized the ability of agents to use, select, and evolve external toolsYaoet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib28)\); Schicket al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib29)\); Fallahpouret al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib30)\)\. Existing studiesShenet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib31)\); Qianet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib32)\)typically focus on identifying the appropriate tool for a given task or decomposing complex tasks into calls to tools with distinct functional roles\. This paradigm has enabled agents to extend their capabilities and solve tasks that require external computation, retrieval, or specialized operationsParanjapeet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib33)\); Wanget al\.\([2025b](https://arxiv.org/html/2605.26691#bib.bib34)\)\.

While effective in many settings, existing tool\-use paradigms often assume that tools are reliable within their intended scope, overlooking failures caused by instance\-level heterogeneity\. This issue is especially critical in medical agents, where tool failures may lead to unsafe downstream decisionsHabliet al\.\([2020](https://arxiv.org/html/2605.26691#bib.bib35)\); Fallahpouret al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib30)\)\. For example, in chest X\-ray diagnosis, a tool that detects cardiomegaly may fail on opacity or pleural effusion, while another tool may exhibit the opposite pattern\. Such non\-overlapping failures stem from heterogeneous clinical instances and tool\-specific biases induced by different architectures and training data\. As a result, single\-tool reliance is risky, but these heterogeneous failure patterns also reveal potential complementarity across tools\. As shown in Figure[1](https://arxiv.org/html/2605.26691#S1.F1)\(a\), each tool uniquely solves instances missed by others, indicating that no single tool is uniformly dominant and that exploiting instance\-level complementarity can recover additional correct predictions beyond the best individual tool\. We visualize the tool’s marginal Oracle111An Oracle is an idealized evaluator that knows the correct answer and is used to estimate the best possible performance among the available candidatesMalmasiet al\.\([2015](https://arxiv.org/html/2605.26691#bib.bib36)\)\.accuracy on datasets CheXpertIrvinet al\.\([2019](https://arxiv.org/html/2605.26691#bib.bib2)\)and MIMIC\-CXRJohnsonet al\.\([2019](https://arxiv.org/html/2605.26691#bib.bib25)\)\. As provided in Figure[1](https://arxiv.org/html/2605.26691#S1.F1)\(b\), the Oracle accuracy increases steadily as more complementary tools are added, substantially exceeding the best single\-tool performance\. This further indicates that the tool set contains instance\-level complementarity and that adding tools with distinct failure patterns can progressively improve the Oracle upper bound\.

Instance\-dependent failure patterns create a gap between the best fixed single tool and an ideal instance\-wise selector\. Prior task\-level tool selection methodsYaoet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib28)\); Qianet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib32)\); Fallahpouret al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib30)\)focus on aggregate tool performance, overlooking instance\-level heterogeneity within the task distribution\. As shown in Figure[1](https://arxiv.org/html/2605.26691#S1.F1), consequently, it fails to fully realize the potential gains indicated by the Oracle performance, motivating an instance\-level policy that exploits tool complementarity for each individual instance\.

Based on these observations, we analyze tool behavior from both aggregate\- and instance\-level perspectives\. We define the Single\-Oracle risk gap to characterize the performance difference between the best single\-tool policy and the Oracle tool set\. Motivated by this risk gap, we propose CSRL, a Collaborative Synergy Reinforcement Learning framework built upon GRPOShaoet al\.\([2024](https://arxiv.org/html/2605.26691#bib.bib27)\)\. CSRL optimizes rewards designed for probabilistic risk minimization and disagreement\-aware synergy learning\. Specifically, we introduce a Brier reward for probabilistic risk minimization under the Brier loss, together with an override reward that encourages correct decisions in disagreement instances and promotes synergy learning\. Furthermore, we introduce Entropy\-Guided Sampling, which constructs heterogeneous training batches by upweighting high\-disagreement instances, thereby providing stronger supervision for instance\-specific synergy\. Experimental results of CSRL show that constructed tool sets can achieve synergistic gains beyond the strongest standalone tool\. Tools with lower average performance may still provide valuable marginal contributions by covering instances missed by stronger tools\. Meanwhile, performance improvements exhibit diminishing returns as the tool set grows, indicating a tradeoff between gains and inference cost\.

![Refer to caption](https://arxiv.org/html/2605.26691v1/fig/figure1-c.png)Figure 1:\(a\) Analysis of tool complementarity, where each number indicates the number of instances correctly identified by the corresponding tools;\(b\) marginal Oracle gains on CheXpert and MIMIC\-CXR;\(c\) Difference between task\-level and instance\-level tool selection\.Our contributions can be summarized as follows:

- •We investigate tool failures in medical agent systems and study how to construct a synergistic set of imperfect tools\. Scalability analyses across tools, models, and datasets provide practical guidelines for tool, dataset, and model selection\.
- •Guided by the Single\-Oracle risk, we develop CSRL, a collaborative synergy reinforcement learning framework that combines entropy\-guided sampling with rewards for probabilistic risk minimization and disagreement\-aware synergy learning, enabling performance beyond the best single tool\.
- •Across two tasks and seven datasets, learned synergy consistently outperforms individual tools, static baselines, and both general\-purpose and specialized multimodal large language models, with average gains of 7\.5% in accuracy and 6\.8% in F1\.

## 2Related Work

### 2\.1Tool\-Augmented Agents and Tool Selection

Recent progress in agentic systems has enabled multimodal large language models \(MLLMs\) to interact with external toolsSchicket al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib29)\); Yaoet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib28)\), APIsPatilet al\.\([2024](https://arxiv.org/html/2605.26691#bib.bib37)\); Liet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib38)\), and specialized modelsShenet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib31)\); Wuet al\.\([2023b](https://arxiv.org/html/2605.26691#bib.bib39)\)to solve tasks beyond their parametric knowledge\. Early work, such as ReActYaoet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib28)\)and ToolformerSchicket al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib29)\), showed that language models can interleave reasoning with tool invocation, while subsequent systems, including HuggingGPTShenet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib31)\), ARTParanjapeet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib33)\), Search\-R1Jinet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib26)\), and ToolRLQianet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib32)\), further explored tool planning, multi\-step execution, and reinforcement learning for tool use\. These methods typically focus on deciding which tool to call for a given task or how to decompose a complex instruction into a sequence of tool invocations\.

### 2\.2Medical Vision\-Language Models and Diagnostic Tools

Medical AI agents have increasingly incorporated external diagnostic models, retrieval systems, and domain\-specific reasoning modules to support clinical question answering, radiology interpretation, triage, and treatment recommendation\(Fallahpouret al\.,[2025](https://arxiv.org/html/2605.26691#bib.bib30)\)\. In chest X\-ray analysis, specialized models such as CheXzeroTiuet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib9)\), MedKLIPWuet al\.\([2023a](https://arxiv.org/html/2605.26691#bib.bib11)\), CheXagentChenet al\.\([2024c](https://arxiv.org/html/2605.26691#bib.bib24)\), and supervised DenseNet variantsCohenet al\.\([2020](https://arxiv.org/html/2605.26691#bib.bib23)\)provide useful diagnostic signals for different diseases and datasetsRajpurkaret al\.\([2017](https://arxiv.org/html/2605.26691#bib.bib40)\)\. More recently, general\-purpose and medical Multimodal Large Language Models \(MLLMs\) have shown promising performance on multimodal medical reasoning tasks\(Chenet al\.,[2024b](https://arxiv.org/html/2605.26691#bib.bib41); Sellergrenet al\.,[2025](https://arxiv.org/html/2605.26691#bib.bib15); Xuet al\.,[2025](https://arxiv.org/html/2605.26691#bib.bib16)\)\.

### 2\.3Reinforcement Learning for Tool\-use Agents

Reinforcement learning has recently been adopted to improve the decision\-making behavior of language\-model agents in settings involving discrete actions, structured outputs, delayed feedback, and non\-differentiable external environments\. In search\-augmented reasoning, policy optimization is used to train agents to issue search queries, incorporate retrieved evidence, and generate final answers through multi\-step interactions\(Jinet al\.,[2025](https://arxiv.org/html/2605.26691#bib.bib26)\)\. In tool\-use settings, reward\-driven optimization further enables language agents to learn when and how to invoke external tools based on task feedback, rather than relying solely on supervised demonstrations or hand\-crafted tool\-use traces\(Qianet al\.,[2025](https://arxiv.org/html/2605.26691#bib.bib32)\)\.

Beyond tool invocation itself, recent RL methods for language models have explored scalable policy optimization algorithms that compare multiple sampled responses for the same input\. Proximal Policy Optimization provides a widely used foundation for stable policy\-gradient training\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.26691#bib.bib42)\), while Group Relative Policy Optimization removes the need for a learned value model by estimating advantages from relative rewards within a group of rollouts\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26691#bib.bib27)\)\. This line of work provides a practical optimization framework for training language agents with rule\-based or task\-specific rewards\.

## 3Method

![Refer to caption](https://arxiv.org/html/2605.26691v1/x1.png)Figure 2:Overview of CSRL\. \(a\) CSRL inference performs instance\-specific agentic tool use\. \(b\) CSRL training combines entropy\-guided sampling with risk\-guided reinforcement learning\.### 3\.1Preliminary

Problem Formulation\.Given a diagnostic sample\(x,q,y\)∼𝒟\(x,q,y\)\\sim\\mathcal\{D\}, wherex∈𝒳x\\in\\mathcal\{X\}is the input image,q∈𝒬q\\in\\mathcal\{Q\}is the target pathology query, andy∈𝒴y\\in\\mathcal\{Y\}is the ground\-truth label, we consider a fixed pool ofKKpretrained diagnostic tools𝒯=\{T1,…,TK\}\\mathcal\{T\}=\\\{T\_\{1\},\\ldots,T\_\{K\}\\\}\. For a given case, the policy may invoke a subset of tools from𝒯\\mathcal\{T\}\. For any invoked toolTkT\_\{k\}, the final output of each tool can be represented aspk=Tk​\(x,q\)∈𝒫p\_\{k\}=T\_\{k\}\(x,q\)\\in\\mathcal\{P\}\. Due to heterogeneity in model architectures and training data, these outputs may exhibit non\-overlapping error patterns\.

To formalize this complementarity, we first define the best single\-tool riskℛsingle⋆\\mathcal\{R\}\_\{\\mathrm\{single\}\}^\{\\star\}as

ℛsingle⋆\\displaystyle\\mathcal\{R\}\_\{\\mathrm\{single\}\}^\{\\star\}=mink∈\[K\]⁡𝔼\(x,q,y\)∼𝒟​\[ℓ​\(pk,y\)\],\\displaystyle=\\min\_\{k\\in\[K\]\}\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[\\ell\(p\_\{k\},y\)\\right\],\(1\)Whereℓ∈ℒ:𝒫×𝒴→𝐑\\ell\\in\\mathcal\{L\}:\\mathcal\{P\}\\times\\mathcal\{Y\}\\rightarrow\\mathbf\{R\}is a diagnostic loss, andℒ\\mathcal\{L\}is the space of loss functions\. As we discussed above, a single best tool is imperfect in real\-world scenarios\. Thus, we consider discovering case\-level tool selection\. Specifically, givenΠ𝒯\\Pi\_\{\\mathcal\{T\}\}be the space of tool integration policyπ:𝒳×𝒬×𝒫K→𝐑\\pi:\\mathcal\{X\}\\times\\mathcal\{Q\}\\times\\mathcal\{P\}^\{K\}\\rightarrow\\mathbf\{R\}, the Oracle risk can be defined as

ℛoracle⋆\\displaystyle\\mathcal\{R\}\_\{\\mathrm\{oracle\}\}^\{\\star\}=infπ∈ΠΘ𝔼\(x,q,y\)∼𝒟​\[ℓ​\(π​\(x,q,𝐩\),y\)\]\.\\displaystyle=\\inf\_\{\\pi\\in\\Pi\_\{\\Theta\}\}\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[\\ell\\\!\\left\(\\pi\(x,q,\\mathbf\{p\}\),y\\right\)\\right\]\.\(2\)Since the single tool strategyπk​\(x,q,𝐩\)=pk\\pi\_\{k\}\(x,q,\\mathbf\{p\}\)=p\_\{k\}is a special case inΠ𝒯\\Pi\_\{\\mathcal\{T\}\}, it is obvious that we can define the Single\-Oracle risk gap as

Δ𝒯=ℛsingle⋆−ℛoracle⋆≥0\.\\Delta\_\{\\mathcal\{T\}\}=\\mathcal\{R\}\_\{\\mathrm\{single\}\}^\{\\star\}\-\\mathcal\{R\}\_\{\\mathrm\{oracle\}\}^\{\\star\}\\geq 0\.\(3\)A positiveΔ𝒯\\Delta\_\{\\mathcal\{T\}\}indicates that the strongest individual tool is not the performance limit of the tool pool\. As shown in the Introduction, performance gains \(Δ𝒯\\Delta\_\{\\mathcal\{T\}\}\) are significantly large, which supports our assumption\.

To optimize the model toward oracle\-level performance, we aim to learn a parameterized tool\-use policyπθ∈Π𝒯\\pi\_\{\\theta\}\\in\\Pi\_\{\\mathcal\{T\}\}\. Givenp^θ=πθ​\(x,q,𝐩\)\\hat\{p\}\_\{\\theta\}=\\pi\_\{\\theta\}\(x,q,\\mathbf\{p\}\), we aim to learn a policy satisfying

ℛoracle⋆≤ℛ​\(πθ\)=𝔼\(x,q,y\)∼𝒟​\[ℓ​\(p^θ,y\)\]<ℛsingle⋆\.\\mathcal\{R\}\_\{\\mathrm\{oracle\}\}^\{\\star\}\\leq\\mathcal\{R\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[\\ell\(\\hat\{p\}\_\{\\theta\},y\)\\right\]<\\mathcal\{R\}\_\{\\mathrm\{single\}\}^\{\\star\}\.\(4\)
To this end, as provided in Figure[2](https://arxiv.org/html/2605.26691#S3.F2), we propose CSRL, a Collaborative Synergy Reinforcement Learning framework built upon GRPOShaoet al\.\([2024](https://arxiv.org/html/2605.26691#bib.bib27)\), which introduces two components that align the optimization procedure with the target in Eq\. \(4\)\. First, we design trajectory\-level rewards for GRPO that encourage valid tool use, minimize probabilistic diagnostic risk, and promote disagreement\-aware synergy recovery, thereby directly pushing downℛ​\(πθ\)\\mathcal\{R\}\(\\pi\_\{\\theta\}\)\. Second, we use Entropy\-Guided Sampling to increase the training mass of high\-disagreement cases, where the gap between single\-tool performance and case\-specific integration is most recoverable\.

#### Tool\-Calling Rollout\.

Rollouts\{τi\}\\\{\\tau\_\{i\}\\\}define the interaction protocol between the policyπθ\\pi\_\{\\theta\}and the tool\-augmented environment: the policyπθ\\pi\_\{\\theta\}uses<tool\_call\>…</tool\_call\>to invoke diagnostic tools, receives tool feedback wrapped by<tool\_response\>…</tool\_response\>, and terminates the trajectory by producing a final prediction inside<answer\>…</answer\>\. The detailed procedure is provided in Appendix[B](https://arxiv.org/html/2605.26691#A2)\.

#### GRPO Optimization\.

We optimizeπθ\\pi\_\{\\theta\}withGroup Relative Policy Optimization\(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2605.26691#bib.bib27)\), using EGS\-stratified mini\-batches \(Sec\.[3\.3](https://arxiv.org/html/2605.26691#S3.SS3)\) and the trajectory\-level rewardRoverallR\_\{\\mathrm\{overall\}\}from Sec\.[3\.2](https://arxiv.org/html/2605.26691#S3.SS2)\. For each prompt\(x,q\)\(x,q\), GRPO samplesGGrollouts\{τi\}i=1G\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{G\}fromπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}and assigns each rollout a group\-relative advantage

A^i=Roverall​\(τi\)−mean​\(\{Roverall​\(τj\)\}j=1G\)std​\(\{Roverall​\(τj\)\}j=1G\),\\hat\{A\}\_\{i\}\\;=\\;\\frac\{R\_\{\\mathrm\{overall\}\}\(\\tau\_\{i\}\)\\,\-\\,\\mathrm\{mean\}\\\!\\left\(\\\{R\_\{\\mathrm\{overall\}\}\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{G\}\\right\)\}\{\\mathrm\{std\}\\\!\\left\(\\\{R\_\{\\mathrm\{overall\}\}\(\\tau\_\{j\}\)\\\}\_\{j=1\}^\{G\}\\right\)\},\(5\)which is shared across all policy\-generated tokens inτi\\tau\_\{i\}\. The GRPO objective combines a PPO\-style clipped surrogate with a token\-level KL penalty to a frozen reference policyπref\\pi\_\{\\mathrm\{ref\}\}:

𝒥GRPO​\(θ\)=𝔼​\[1G​∑i=1G1\|τi\|​∑t=1\|τi\|\{min⁡\(ρi,t​A^i,clip​\(ρi,t,1−ϵ,1\+ϵ\)​A^i\)−β​𝔻KL​\[πθ∥πref\]\}\],\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)\\;=\\;\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|\\tau\_\{i\}\|\}\\sum\_\{t=1\}^\{\|\\tau\_\{i\}\|\}\\Big\\\{\\min\\\!\\big\(\\rho\_\{i,t\}\\hat\{A\}\_\{i\},\\;\\mathrm\{clip\}\(\\rho\_\{i,t\},\\,1\\\!\-\\\!\\epsilon,\\,1\\\!\+\\\!\\epsilon\)\\,\\hat\{A\}\_\{i\}\\big\)\-\\beta\\,\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\\\!\\big\[\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\\big\]\\Big\\\}\\right\],\(6\)with importance ratio

ρi,t=πθ​\(ai,t∣si,t\)πθold​\(ai,t∣si,t\)\.\\rho\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(a\_\{i,t\}\\\!\\mid\\\!s\_\{i,t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{i,t\}\\\!\\mid\\\!s\_\{i,t\}\)\}\.\(7\)

### 3\.2Risk\-guided Reward Design

To train the policyπθ\\pi\_\{\\theta\}with GRPO, we assign each rollout a rule\-based reward composed of three terms that capture format validity, probabilistic correctness, and case\-specific majority override:

Roverall=Rbrier\+Roverride\+Rformat\.R\_\{\\mathrm\{overall\}\}=R\_\{\\mathrm\{brier\}\}\+R\_\{\\mathrm\{override\}\}\+R\_\{\\mathrm\{format\}\}\.\(8\)
To encourage a lowℛ​\(πθ\)\\mathcal\{R\}\(\\pi\_\{\\theta\}\)small, we first introduce the Brier reward as follows:

Rbrier​\(p^θ,y\)=\{1−\(p^θ−y\)2,if​p^θ∈\[0,1\]​is parseable and​nvalid\>0,0,otherwise\.R\_\{\\mathrm\{brier\}\}\(\\hat\{p\}\_\{\\theta\},y\)=\\begin\{cases\}1\-\(\\hat\{p\}\_\{\\theta\}\-y\)^\{2\},&\\begin\{array\}\[\]\{l\}\\text\{if \}\\hat\{p\}\_\{\\theta\}\\in\[0,1\]\\text\{ is parseable and \}n\_\{\\mathrm\{valid\}\}\>0,\\end\{array\}\\\\\[5\.0pt\] 0,&\\text\{otherwise\}\.\\end\{cases\}\(9\)Where,nvalidn\_\{\\mathrm\{valid\}\}denote the number of valid tool responses observed in the rollout\. The Brier reward evaluates probabilistic correctness only when the rollout uses valid tool evidence\. The following proposition characterizes the role ofRbrierR\_\{\\mathrm\{brier\}\}

###### Proposition 3\.1\.

Assume that the prediction risk is defined with the Brier loss, i\.e\.,

ℛ​\(πθ\)=𝔼\(x,q,y\)∼𝒟​\[\(p^θ−y\)2\]\.\\mathcal\{R\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[\(\\hat\{p\}\_\{\\theta\}\-y\)^\{2\}\\right\]\.Then maximizing the Brier reward is equivalent to minimizing the prediction risk:

arg⁡maxθ⁡𝔼​\[Rbrier​\(p^θ,y\)\]=arg⁡minθ⁡ℛ​\(πθ\)\.\\arg\\max\_\{\\theta\}\\mathbb\{E\}\\left\[R\_\{\\mathrm\{brier\}\}\(\\hat\{p\}\_\{\\theta\},y\)\\right\]=\\arg\\min\_\{\\theta\}\\mathcal\{R\}\(\\pi\_\{\\theta\}\)\.

Proposition[3\.1](https://arxiv.org/html/2605.26691#S3.Ex2)illustrates that maximizing rewardRbrierR\_\{\\mathrm\{brier\}\}is aligned with minimizing the riskℛ​\(πθ\)\\mathcal\{R\}\(\\pi\_\{\\theta\}\)\.RbrierR\_\{\\mathrm\{brier\}\}provides a proper probabilistic training signal for the final diagnostic score, but it is agnostic to the structure of tool disagreement\. Such disagreement is central to the synergy gap, as it indicates that different tools succeed and fail on different cases, leaving room for a case\-specific policy to recover information that no fixed tool can consistently capture\.

To address the above issue, we introduce the override reward in the following\.

Roverride​\(p^θ,m,y\)=\{α,if​ℓ​\(p^θ,y\)<s​and​ℓ​\(p^m,y\)≥s,−α,if​ℓ​\(p^θ,y\)≥s​and​ℓ​\(p^m,y\)<s,0,otherwise\.R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)=\\begin\{cases\}\\alpha,&\\text\{if \}\\ell\(\\hat\{p\}\_\{\\theta\},y\)<s\\text\{ and \}\\ell\(\\hat\{p\}\_\{m\},y\)\\geq s,\\\\ \-\\alpha,&\\text\{if \}\\ell\(\\hat\{p\}\_\{\\theta\},y\)\\geq s\\text\{ and \}\\ell\(\\hat\{p\}\_\{m\},y\)<s,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(10\)Wherep^m\\hat\{p\}\_\{m\}denotes the response of the majority vote among the valid invoked tools,ssis a threshold used to determine whether a prediction is consistent with the ground\-truth labelyybased on the lossℓ\\ell, e\.g\.,s=0\.5s=0\.5in classification tasks, andα\\alphais the reward value which is set as 0\.1 in our experiments\. The following proposition demonstrates howRoverrideR\_\{\\mathrm\{override\}\}enlarges the risk gap\.

###### Proposition 3\.2\.

Considering the riskℛ​\(πθ\)=Pr⁡\(p^θ≠y\)\\mathcal\{R\}\(\\pi\_\{\\theta\}\)=\\Pr\(\\hat\{p\}\_\{\\theta\}\\neq y\)and the risk gap𝒢​\(πθ\)=ℛsingle⋆−ℛ​\(πθ\)\\mathcal\{G\}\(\\pi\_\{\\theta\}\)=\\mathcal\{R\}^\{\\star\}\_\{\\mathrm\{single\}\}\-\\mathcal\{R\}\(\\pi\_\{\\theta\}\), maximizingRoverride​\(p^θ,m,y\)R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)is aligned with maximizing𝒢​\(πθ\)\\mathcal\{G\}\(\\pi\_\{\\theta\}\)\.

Proposition[3\.2](https://arxiv.org/html/2605.26691#S3.Thmtheorem2)shows that the override reward is aligned with our objective of increasing𝒢​\(πθ\)\\mathcal\{G\}\(\\pi\_\{\\theta\}\), thereby improving the policy beyond the best single\-tool baseline\.

In addition, to encourage the policy to follow the required structured output schema, we introduce a format reward as follows:

Rformat=\{α,if all required fields appear in the correct orderand the terminal answer is parseable into​\[0,1\],0,otherwise\.R\_\{\\mathrm\{format\}\}=\\begin\{cases\}\\alpha,&\\begin\{array\}\[\]\{l\}\\text\{if all required fields appear in the correct order\}\\\\ \\text\{and the terminal answer is parseable into \}\[0,1\],\\end\{array\}\\\\\[5\.0pt\] 0,&\\text\{otherwise\}\.\\end\{cases\}\(11\)In our experiments, we setα=0\.1\\alpha=0\.1inRformatR\_\{\\mathrm\{format\}\}\. Although we present the reward design under a classification setting in this section, it can be readily adapted to regression tasks, which we discuss in the Experiments section\.

### 3\.3Entropy\-Guided Sampling

The risk gapΔ𝒯\\Delta\_\{\\mathcal\{T\}\}is most visible in cases where the tool pool provides heterogeneous evidence\. Under uniform sampling from the natural data distribution, these high\-disagreement cases may be underrepresented, causing most GRPO groups to be dominated by consensus cases with weak signals for learning case\-specific tool synergy\. We therefore introduce Entropy\-Guided Sampling \(EGS\), a static*per\-batch*stratified sampler that reweights mini\-batch construction toward high\-entropy regions of the tool\-evidence space\. By increasing the frequency of disagreement cases during training, EGS provides denser policy\-gradient signal for learning when and how to exploit complementary tools\.

For each training case\(x,q,y\)\(x,q,y\), we record the number of correctly classifying tools:

nc​\(x,q\)=∑k=1K𝕀​\{ℓ​\(p^k,y\)<s\},n\_\{c\}\(x,q\)=\\sum\_\{k=1\}^\{K\}\\mathbb\{I\}\\\{\\ell\(\\hat\{p\}\_\{k\},y\)<s\\\},\(12\)wheressis the threshold discussed in Eq\.[10](https://arxiv.org/html/2605.26691#S3.E10)\. We then measure the correctness\-agreement entropy by

H​\(n¯\)=−n¯​log⁡n¯−\(1−n¯\)​log⁡\(1−n¯\),n¯=ncK\.H\(\\bar\{n\}\)=\-\\bar\{n\}\\log\\bar\{n\}\-\(1\-\\bar\{n\}\)\\log\(1\-\\bar\{n\}\),\\qquad\\bar\{n\}=\\frac\{n\_\{c\}\}\{K\}\.\(13\)This entropy quantifies case\-level tool disagreement, reaching its maximum atnc=K/2n\_\{c\}=K/2and vanishing atnc∈\{0,K\}n\_\{c\}\\in\\\{0,K\\\}\. The useful learning signal for case\-specific synergy is concentrated in the conflict region1≤nc≤K−11\\leq n\_\{c\}\\leq K\-1, where correct and incorrect tools coexist\. In contrast, consensus cases provide limited synergy signal:nc=0n\_\{c\}=0suggests either label noise or cases outside the capability of the tool pool, whereasnc=Kn\_\{c\}=Kindicates that all tools are already correct and case\-specific aggregation cannot further improve the prediction\.

We therefore stratify the training set according to the entropy\-induced disagreement level\. Let\{𝒮j\}j=1K′\\\{\\mathcal\{S\}\_\{j\}\\\}\_\{j=1\}^\{K^\{\\prime\}\}denote the resulting subsets of all data, ordered by increasing entropy\. During mini\-batch construction, each subset𝒮j\\mathcal\{S\}\_\{j\}is assigned a sampling probabilityρj\\rho\_\{j\}proportional to its entropy level\. Since the entropy ofS1S\_\{1\}is obviously 0, we assign a small probability to this subset exclusively\. Consequently, high\-entropy subsets, which contain stronger tool disagreement and richer synergy signals, are sampled more frequently than low\-entropy consensus subsets\. This entropy\-proportional sampling scheme biases training toward cases where case\-specific tool integration is most beneficial, while preserving a nonzero sampling probability for all subsets to maintain calibration on easy or consensus cases\.

## 4Experiments

### 4\.1EXPERIMENTAL SETUP

Datasets\.We conduct experiments on two tasks and seven datasets on the chest X\-ray \(CXR\) modality\. For*binary pathology classification*\(Task 1\), CSRL is trained on the official training splits of CheXpertIrvinet al\.\([2019](https://arxiv.org/html/2605.26691#bib.bib2)\)and MIMIC\-CXRJohnsonet al\.\([2019](https://arxiv.org/html/2605.26691#bib.bib25)\), and evaluated on six test benchmarks: their held\-out test splits, three stratified1,0001\{,\}000\-image out\-of\-domain samples from NIH ChestX\-ray14Wanget al\.\([2017](https://arxiv.org/html/2605.26691#bib.bib3)\), RSNA Pneumonia DetectionShihet al\.\([2019](https://arxiv.org/html/2605.26691#bib.bib4)\), and VinDr\-CXRNguyenet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib5)\), plus NIH\-GoogleMajkowskaet al\.\([2020](https://arxiv.org/html/2605.26691#bib.bib6)\)\(1,9621\{,\}962radiologist\-adjudicated images\) as a clean\-label check against NIH NLP label noise\. For*medical VQA generalization*\(Task 2\), the policy is trained on a stratified5,0005\{,\}000\-sample subset of thechoose\_trainsplit of MIMIC\-Ext\-MIMIC\-CXR\-VQABaeet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib7)\)under the M4CXR non\-empty\-GT setting, and evaluated on the gold1,5231\{,\}523\-question test split and its933933\-question non\-empty\-GT subsetParket al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib8)\)\.

Policy Model and Tool Pools\.We use Qwen2\.5\-VL\-7B\-InstructBaiet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib19)\)as the policy model for reinforcement learning\. We assemble two task\-specific tool pools chosen for diversity in training data and architecture\. For Task 1 \(*binary pathology classification*\), the tool pool includes six specialist tools, namely CheXagentChenet al\.\([2024c](https://arxiv.org/html/2605.26691#bib.bib24)\), CheXzeroTiuet al\.\([2022](https://arxiv.org/html/2605.26691#bib.bib9)\), MedKLIPWuet al\.\([2023a](https://arxiv.org/html/2605.26691#bib.bib11)\), and three DenseNet\-121 backbones from torchxrayvisionCohenet al\.\([2020](https://arxiv.org/html/2605.26691#bib.bib23)\)\. For Task 2 \(*VQA generalization*\), the tool pool includes three medical vision\-language models, namely CheXagent\-8BChenet al\.\([2024c](https://arxiv.org/html/2605.26691#bib.bib24)\), RadVLM\-7BDeperroiset al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib22)\), and MedGemma\-4B\-ITSellergrenet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib15)\)\.

Implementation Details\.We implement CSRL with the Qwen2\.5\-VL\-7B\-InstructBaiet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib19)\)backbone\. The policy is optimized using the Group Relative Policy Optimization \(GRPO\) algorithmShaoet al\.\([2024](https://arxiv.org/html/2605.26691#bib.bib27)\); Guoet al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib21)\)for∼75\\sim\\\!75iterations on 8 NVIDIA H200 GPUs, built onVeRLShenget al\.\([2025](https://arxiv.org/html/2605.26691#bib.bib20)\)with FSDP training and asynchronous vLLMKwonet al\.\([2023](https://arxiv.org/html/2605.26691#bib.bib10)\)rollouts\. During training, we use a batch size of256256prompts; for each prompt, the agent samples1616trajectories with up to44reasoning turns and a maximum of66parallel tool calls per turn\. Key hyperparameters are AdamW with learning rate1×10−61\{\\times\}10^\{\-6\}, PPO clip ratio0\.20\.2, low\-variance KL regularization with coefficient10−310^\{\-3\}against a frozen reference policy, and an entropy bonus of10−310^\{\-3\}\.

Baselines and Comparisons\.We compare CSRL with a broad set of baselines spanning four categories: open\-source general vision\-language models \(VLMs\), open\-source medical MLLMs, closed\-source general MLLMs, and specialist CXR diagnostic tools\. To further assess whether the gains come from case\-specific synergy rather than simply using multiple tools, we include several tool\-combination baselines, including logistic regression, majority voting, and best\-tool\-per\-pathology selection\.

### 4\.2Experimental Analysis

Table 1:Performance comparison for binary disease diagnosis across six chest X\-ray benchmarks\. We report macro accuracy \(Acc\) and macro F1\. Out\-of\-domain \(OOD\) test sets are marked with⋄\\diamond\. Best results are highlighted inbold, and second\-best results areunderlined\.Table 2:Ablation study on three CXR benchmarks\. We report macro accuracy \(Acc\) and macro F1\.⋄\\diamond: OOD test sets\. “w\.nntool\(s\)”: allow at mostnndiagnostic tools per rollout\. All models are trained on CheXpert and MIMIC\-CXR, and all changes are measured relative to CSRL\.
To systematically evaluate the effectiveness of our proposed approach, we organize the experiment as follows\.

Learning tool synergy yields robust gains across in\-domain and out\-of\-domain datasets\.Our

Table 3:Generalization performance on the VQA task\. We report F1, exact match \(EM\), precision \(P\), recall \(R\), and their average score\. Best results are highlighted inbold\.
proposed CSRL achieves the best overall performance among all compared methods \(see Table[4\.2](https://arxiv.org/html/2605.26691#S4.SS2)\), obtaining an average Acc of 0\.822 and an average F1 of 0\.578 across the six CXR benchmarks, outperforming the strongest prior method by 3\.3% in accuracy and 4\.8% in F1 score\. Compared with the strongest individual tool, CheXzero, CSRL improves the average performance by7\.5%in Acc and6\.8%in F1, showing that explicitly learning tool synergy is more effective than relying on any single tool\. This advantage is also consistent across datasets: CSRL almost achieves the best or the second\-best F1 score and Acc on all six benchmarks\. Notably, CSRL performs strongly on both in\-domain and out\-of\-domain test sets, including ChestX\-ray14, VinDr\-CXR, NIH\-Google, and RSNA, suggesting that learned tool synergy helps the model better exploit complementary diagnostic capabilities under distribution shifts\. In contrast, individual VLMs, specialist CXR tools, and simple combination baselines often perform well only on specific datasets or metrics, but their average performance remains lower\. These results demonstrate that CSRL can learn when and how different tools complement each other, leading to more robust and accurate binary disease diagnosis than the best individual tool\.

Beyond binary disease diagnosis, Table[3](https://arxiv.org/html/2605.26691#S4.T3)further evaluates the generalization ability of CSRL on the VQA task\. CSRL consistently achieves the best performance across all metrics\. Compared with the strongest baseline, MedGemma\-4B\-IT, CSRL improves the average score from 0\.657 to 0\.704, corresponding to a gain of4\.7%, indicating that learning tool synergy not only improves classification performance but also enhances the model’s ability to recover more complete and accurate answers in a different task setting\.

Trade\-off between tool\-set size, performance, and cost\.Figure[3](https://arxiv.org/html/2605.26691#S4.F3)reveals a clear trade\-off between tool\-set size, performance, and computational cost\. Increasing the number of tools generally improves performance, but the marginal gain diminishes as the tool set becomes larger\. As shown in the tool scalability ablation \(Figure[3](https://arxiv.org/html/2605.26691#S4.F3)\), expanding the tool set from T1 to T3 brings a substantial improvement, with average macro Acc increasing from 70\.29% to 77\.60% and average macro F1 increasing from 52\.40% to 54\.25%\. However, further increasing the tool set from T3 to T6 yields only limited additional gains, with Acc fluctuating around 77%–78% and F1 improving more moderately to 55\.82%\. This suggests that a small number of complementary tools can already capture most of the useful inter\-tool synergy, while adding more tools may introduce redundancy and higher inference cost\.

The results also show that performance is affected not only by the number of tools, but also by the capacity of the policy model \(Figure[3](https://arxiv.org/html/2605.26691#S4.F3)\) and the quality of training data \(Figures[3](https://arxiv.org/html/2605.26691#S4.F3)and[3](https://arxiv.org/html/2605.26691#S4.F3)\)\. Larger policy models consistently improve performance, from 72\.31% Acc and 52\.85% F1 with the 2B model to 77\.76% Acc and 55\.82% F1 with the 7B model, indicating that stronger policy models are better at learning when different tools should be trusted or combined\. Similarly, increasing the training data scale improves both Acc and F1, while using a more diverse training composition further strengthens generalization\. Overall, these results indicate that the best performance\-cost balance is achieved by selecting a compact but complementary tool set and pairing it with a sufficiently capable policy model, rather than naively increasing the number of tools\.

### 4\.3Ablation Study

Table[2](https://arxiv.org/html/2605.26691#S4.T2)presents a comprehensive ablation study of CSRL\. Overall, the full CSRL model achieves the best average performance, with an average Acc of 0\.778 and an average F1 of 0\.558, demonstrating the effectiveness of each major design component\. First, reducing the policy model scale consistently degrades performance\. Using Qwen3\-VL\-2B or Qwen2\.5\-VL\-3B as the policy model lowers the average Acc by 5\.5% and 3\.3%, respectively, indicating that a stronger policy model is important for learning reliable tool synergy\. Second, reducing the training data scale also harms performance: using only 10% and 50% of the training data decreases average Acc by 2\.8% and 0\.9%, and average F1 by 4\.0% and 1\.4%, respectively\. This suggests that sufficient training data helps the policy model better learn when and how to combine diagnostic tools\.

![Refer to caption](https://arxiv.org/html/2605.26691v1/x2.png)

Figure 3:Scalability studies of CSRL\. \(a\) Scalability across training data ratios; \(b\) Scalability across different dataset compositions; \(c\) Scalability across various policy models\. \(d\) Scalability across tool\-set sizes\. T1\-T6 denote tool sets with 1 to 6 tools, respectively\.\\phantomsubcaption

\\phantomsubcaption

\\phantomsubcaption

\\phantomsubcaption

The ablation on training dataset composition further shows that dataset diversity contributes to generalization\. Training only on CheXpert or additionally including ChestX\-ray14 both lead to lower average performance than the default CSRL setting, suggesting that simply adding more data is not always sufficient; the training composition must also align with the target generalization setting\. The sampling and reward design ablation confirms the importance of both proposed components\. Removing entropy\-guided sampling decreases the average Acc and F1 by 2\.2% and 4\.5%, while removing the override reward reduces them by 0\.9% and 1\.9%, showing that exploration and disagreement\-aware reward shaping are both beneficial for recovering tool synergy\. Finally, the tool\-call budget ablation reveals a clear performance\-cost trade\-off\. Allowing only one or two tools substantially reduces performance, while using three to five tools approaches the full model\. This indicates that multiple tools are necessary to capture complementary diagnostic signals, but most of the benefit can be obtained with a moderate tool budget\. Together, these results validate the contribution of the policy model, training data, reward design, and tool budget in enabling CSRL to learn robust tool synergy\.

## 5Conclusion and Limitations

Motivated by the gap between task\-level and case\-level tool selection, we presented CSRL \(Collaborative Synergy Reinforcement Learning\), a GRPO\-based framework for learning instance\-level synergy among imperfect medical tools\. Across chest X\-ray diagnosis and medical VQA, CSRL consistently outperforms individual tools, static combinations, and general\-purpose or medical multimodal models, showing that synergistic gains arise from case\-level complementarity rather than standalone accuracy\.Limitations\.Our study focuses on chest X\-ray benchmarks, predefined tool pools, and retrospective evaluation\. Future work will extend CSRL to broader clinical tasks, dynamic tool ecosystems, and prospective human\-in\-the\-loop validation before deployment\.

## References

- \[1\]S\. Bae, D\. Kyung, J\. Ryu, E\. Cho, G\. Lee, S\. Kweon, J\. Oh, L\. Ji, E\. Chang, T\. Kim,et al\.\(2023\)Ehrxqa: a multi\-modal question answering dataset for electronic health records with chest x\-ray images\.Advances in Neural Information Processing Systems36,pp\. 3867–3880\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[2\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin\(2025\)Qwen2\.5\-vl technical report\.ArXivabs/2502\.13923\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276449796)Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p3.9),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.22.2.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.24.4.1)\.
- \[3\]J\. Chen, C\. Gui, R\. Ouyang, A\. Gao, S\. Chen, G\. H\. Chen, X\. Wang, Z\. Cai, K\. Ji, X\. Wan,et al\.\(2024\)Towards injecting medical visual knowledge into multimodal llms at scale\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 7346–7370\.Cited by:[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.26.6.1)\.
- \[4\]J\. Chen, R\. Ouyang, A\. Gao, S\. Chen, G\. H\. Chen, X\. Wang, R\. Zhang, Z\. Cai, K\. Ji, G\. Yu, X\. Wan, and B\. Wang\(2024\)HuatuoGPT\-vision, towards injecting medical visual knowledge into multimodal llms at scale\.External Links:2406\.19280,[Link](https://arxiv.org/abs/2406.19280)Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1)\.
- \[5\]Z\. Chen, M\. Varma, J\. Delbrouck, M\. Paschali, L\. Blankemeier, D\. Van Veen, J\. M\. J\. Valanarasu, A\. Youssef, J\. P\. Cohen, E\. P\. Reis,et al\.\(2024\)Chexagent: towards a foundation model for chest x\-ray interpretation\.InAAAI 2024 Spring Symposium on Clinical Foundation Models,Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1)\.
- \[6\]Z\. Chen, M\. Varma, J\. Xu, M\. Paschali, D\. Van Veen, A\. Johnston, A\. Youssef, L\. Blankemeier, C\. Bluethgen, S\. Altmayer,et al\.\(2024\)A vision\-language foundation model to enhance efficiency of chest x\-ray interpretation\.arXiv preprint arXiv:2401\.12208\.Cited by:[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.40.20.1)\.
- \[7\]J\. P\. Cohen, J\. D\. Viviano, P\. Bertin, P\. Morrison, P\. Torabian, M\. Guarrera, M\. P\. Lungren, A\. Chaudhari, R\. Brooks, M\. Hashir,et al\.\(2022\)TorchXRayVision: a library of chest x\-ray datasets and models\.InInternational Conference on Medical Imaging with Deep Learning,pp\. 231–249\.Cited by:[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.35.15.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.36.16.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.37.17.1)\.
- \[8\]J\. P\. Cohen, J\. Viviano, M\. Hashir, and H\. Bertrand\(2020\)TorchXRayVision: a library of chest x\-ray datasets and models \(2020\)\.URL https://arxiv\. org/abs/2111\.00595\.Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1)\.
- \[9\]N\. Deperrois, H\. Matsuo, S\. Ruipérez\-Campillo, M\. Vandenhirtz, S\. Laguna, A\. Ryser, K\. Fujimoto, M\. Nishio, T\. M\. Sutter, J\. E\. Vogt,et al\.\(2025\)RadVLM: a multitask conversational vision\-language model for radiology\.arXiv preprint arXiv:2502\.03333\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1)\.
- \[10\]A\. Fallahpour, J\. Ma, A\. Munim, H\. Lyu, and B\. Wang\(2025\)Medrax: medical reasoning agent for chest x\-ray\.arXiv preprint arXiv:2502\.02673\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1),[§1](https://arxiv.org/html/2605.26691#S1.p2.1),[§1](https://arxiv.org/html/2605.26691#S1.p3.1),[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1)\.
- \[11\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p3.9)\.
- \[12\]I\. Habli, T\. Lawton, and Z\. Porter\(2020\)Artificial intelligence in health care: accountability and safety\.Bulletin of the World Health Organization98\(4\),pp\. 251\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p2.1)\.
- \[13\]J\. Irvin, P\. Rajpurkar, M\. Ko, Y\. Yu, S\. Ciurea\-Ilcus, C\. Chute, H\. Marklund, B\. Haghgoo, R\. Ball, K\. Shpanskaya,et al\.\(2019\)Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison\.InProceedings of the AAAI conference on artificial intelligence,Vol\.33,pp\. 590–597\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[14\]S\. Jiang, Y\. Wang, S\. Song, T\. Hu, C\. Zhou, B\. Pu, Y\. Zhang, Z\. Yang, Y\. Feng, J\. T\. Zhou,et al\.\(2025\)Hulu\-med: a transparent generalist model towards holistic medical vision\-language understanding\.arXiv preprint arXiv:2510\.08668\.Cited by:[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.29.9.1)\.
- \[15\]B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. Han\(2025\)Search\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2605.26691#S2.SS3.p1.1)\.
- \[16\]A\. E\. Johnson, T\. J\. Pollard, S\. J\. Berkowitz, N\. R\. Greenbaum, M\. P\. Lungren, C\. Deng, R\. G\. Mark, and S\. Horng\(2019\)MIMIC\-cxr, a de\-identified publicly available database of chest radiographs with free\-text reports\.Scientific data6\(1\),pp\. 317\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[17\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p3.9)\.
- \[18\]M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li\(2023\)Api\-bank: a comprehensive benchmark for tool\-augmented llms\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 3102–3116\.Cited by:[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.
- \[19\]A\. Majkowska, S\. Mittal, D\. F\. Steiner, J\. J\. Reicher, S\. M\. McKinney, G\. E\. Duggan, K\. Eswaran, P\. Cameron Chen, Y\. Liu, S\. R\. Kalidindi,et al\.\(2020\)Chest radiograph interpretation with deep learning models: assessment with radiologist\-adjudicated reference standards and population\-adjusted evaluation\.Radiology294\(2\),pp\. 421–431\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[20\]S\. Malmasi, J\. Tetreault, and M\. Dras\(2015\)Oracle and human baselines for native language identification\.InProceedings of the tenth workshop on innovative use of NLP for building educational applications,pp\. 172–178\.Cited by:[footnote 1](https://arxiv.org/html/2605.26691#footnote1)\.
- \[21\]H\. Q\. Nguyen, K\. Lam, L\. T\. Le, H\. H\. Pham, D\. Q\. Tran, D\. B\. Nguyen, D\. D\. Le, C\. M\. Pham, H\. T\. Tong, D\. H\. Dinh,et al\.\(2022\)VinDr\-cxr: an open dataset of chest x\-rays with radiologist’s annotations\.Scientific Data9\(1\),pp\. 429\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[22\]B\. Paranjape, S\. Lundberg, S\. Singh, H\. Hajishirzi, L\. Zettlemoyer, and M\. T\. Ribeiro\(2023\)Art: automatic multi\-step reasoning and tool\-use for large language models\.arXiv preprint arXiv:2303\.09014\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.
- \[23\]J\. Park, S\. Kim, B\. Yoon, J\. Hyun, and K\. Choi\(2025\)M4CXR: exploring multitask potentials of multimodal large language models for chest x\-ray interpretation\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[24\]S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez\(2024\)Gorilla: large language model connected with massive APIs\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=tBRNC6YemY)Cited by:[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.
- \[25\]C\. Qian, E\. C\. Acikgoz, Q\. He, H\. Wang, X\. Chen, D\. Hakkani\-Tür, G\. Tur, and H\. Ji\(2025\)Toolrl: reward is all tool learning needs\.arXiv preprint arXiv:2504\.13958\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1),[§1](https://arxiv.org/html/2605.26691#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2605.26691#S2.SS3.p1.1)\.
- \[26\]P\. Rajpurkar, J\. Irvin, K\. Zhu, B\. Yang, H\. Mehta, T\. Duan, D\. Ding, A\. Bagul, C\. Langlotz, K\. Shpanskaya,et al\.\(2017\)Chexnet: radiologist\-level pneumonia detection on chest x\-rays with deep learning\.arXiv preprint arXiv:1711\.05225\.Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1)\.
- \[27\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.Advances in neural information processing systems36,pp\. 68539–68551\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.
- \[28\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.3](https://arxiv.org/html/2605.26691#S2.SS3.p2.1)\.
- \[29\]A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.\(2025\)Medgemma technical report\.arXiv preprint arXiv:2507\.05201\.Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.28.8.1)\.
- \[30\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p4.1),[§2\.3](https://arxiv.org/html/2605.26691#S2.SS3.p2.1),[§3\.1](https://arxiv.org/html/2605.26691#S3.SS1.SSS0.Px2.p1.6),[§3\.1](https://arxiv.org/html/2605.26691#S3.SS1.p4.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p3.9)\.
- \[31\]Y\. Shen, K\. Song, X\. Tan, D\. Li, W\. Lu, and Y\. Zhuang\(2023\)Hugginggpt: solving ai tasks with chatgpt and its friends in hugging face\.Advances in Neural Information Processing Systems36,pp\. 38154–38180\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.
- \[32\]G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu\(2025\)Hybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p3.9)\.
- \[33\]G\. Shih, C\. C\. Wu, S\. S\. Halabi, M\. D\. Kohli, L\. M\. Prevedello, T\. S\. Cook, A\. Sharma, J\. K\. Amorosa, V\. Arteaga, M\. Galperin\-Aizenberg,et al\.\(2019\)Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia\.Radiology: Artificial Intelligence1\(1\),pp\. e180041\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[34\]E\. Tiu, E\. Talius, P\. Patel, C\. P\. Langlotz, A\. Y\. Ng, and P\. Rajpurkar\(2022\)Expert\-level detection of pathologies from unannotated chest x\-ray images via self\-supervised learning\.Nature biomedical engineering6\(12\),pp\. 1399–1406\.Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.38.18.1)\.
- \[35\]W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.\(2025\)Internvl3\. 5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.arXiv preprint arXiv:2508\.18265\.Cited by:[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.23.3.1)\.
- \[36\]X\. Wang, Y\. Peng, L\. Lu, Z\. Lu, M\. Bagheri, and R\. M\. Summers\(2017\)Chestx\-ray8: hospital\-scale chest x\-ray database and benchmarks on weakly\-supervised classification and localization of common thorax diseases\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 2097–2106\.Cited by:[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p1.5)\.
- \[37\]X\. Wang, B\. Wang, D\. Lu, J\. Yang, T\. Xie, J\. Wang, J\. Deng, X\. Guo, Y\. Xu, C\. H\. Wu,et al\.\(2025\)Opencua: open foundations for computer\-use agents\.arXiv preprint arXiv:2508\.09123\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1)\.
- \[38\]C\. Wu, X\. Zhang, Y\. Zhang, Y\. Wang, and W\. Xie\(2023\)Medklip: medical knowledge enhanced language\-image pre\-training in radiology\.arXiv preprint arXiv:2301\.02228\.Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.26691#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.39.19.1)\.
- \[39\]C\. Wu, S\. Yin, W\. Qi, X\. Wang, Z\. Tang, and N\. Duan\(2023\)Visual chatgpt: talking, drawing and editing with visual foundation models\.arXiv preprint arXiv:2303\.04671\.Cited by:[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.
- \[40\]W\. Xu, H\. P\. Chan, L\. Li, M\. Aljunied, R\. Yuan, J\. Wang, C\. Xiao, G\. Chen, C\. Liu, Z\. Li,et al\.\(2025\)Lingshu: a generalist foundation model for unified multimodal medical understanding and reasoning\.arXiv preprint arXiv:2506\.07044\.Cited by:[§2\.2](https://arxiv.org/html/2605.26691#S2.SS2.p1.1),[§4\.2](https://arxiv.org/html/2605.26691#S4.SS2.22.22.20.27.7.1)\.
- \[41\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§1](https://arxiv.org/html/2605.26691#S1.p1.1),[§1](https://arxiv.org/html/2605.26691#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.26691#S2.SS1.p1.1)\.

## Appendix ATheoretical Proof

###### Proof of Proposition[3\.1](https://arxiv.org/html/2605.26691#S3.Ex2)\.

By definition, the Brier reward is

Rbrier​\(p^θ,y\)\\displaystyle R\_\{\\mathrm\{brier\}\}\(\\hat\{p\}\_\{\\theta\},y\)=1−\(p^θ−y\)2\.\\displaystyle=1\-\(\\hat\{p\}\_\{\\theta\}\-y\)^\{2\}\.Taking expectation over\(x,q,y\)∼𝒟\(x,q,y\)\\sim\\mathcal\{D\}, we obtain

𝔼\(x,q,y\)∼𝒟​\[Rbrier​\(p^θ,y\)\]\\displaystyle\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[R\_\{\\mathrm\{brier\}\}\(\\hat\{p\}\_\{\\theta\},y\)\\right\]=𝔼\(x,q,y\)∼𝒟​\[1−\(p^θ−y\)2\]\\displaystyle=\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[1\-\(\\hat\{p\}\_\{\\theta\}\-y\)^\{2\}\\right\]=1−𝔼\(x,q,y\)∼𝒟​\[\(p^θ−y\)2\]\\displaystyle=1\-\\mathbb\{E\}\_\{\(x,q,y\)\\sim\\mathcal\{D\}\}\\left\[\(\\hat\{p\}\_\{\\theta\}\-y\)^\{2\}\\right\]=1−ℛ​\(πθ\)\.\\displaystyle=1\-\\mathcal\{R\}\(\\pi\_\{\\theta\}\)\.Therefore,

arg⁡maxθ⁡𝔼​\[Rbrier​\(p^θ,y\)\]=arg⁡minθ⁡ℛ​\(πθ\)\.\\arg\\max\_\{\\theta\}\\mathbb\{E\}\\left\[R\_\{\\mathrm\{brier\}\}\(\\hat\{p\}\_\{\\theta\},y\)\\right\]=\\arg\\min\_\{\\theta\}\\mathcal\{R\}\(\\pi\_\{\\theta\}\)\.∎

###### proof of Proposition[3\.2](https://arxiv.org/html/2605.26691#S3.Thmtheorem2)\.

Accoring to the definition ofRoverrideR\_\{\\mathrm\{override\}\}, the decision made bysscan be presented as

Pr⁡\(p=y\)=Pr⁡\(ℓ​\(p,y\)<s\),Pr⁡\(p≠y\)=Pr⁡\(ℓ​\(p,y\)≥s\)\\Pr\(p=y\)=\\Pr\(\\ell\(p,y\)<s\),\\Pr\(p\\neq y\)=\\Pr\(\\ell\(p,y\)\\geq s\)Then we can define the risk of the policy as

ℛ​\(πθ\)=Pr⁡\(p^θ≠y\),\\mathcal\{R\}\(\\pi\_\{\\theta\}\)=\\Pr\(\\hat\{p\}\_\{\\theta\}\\neq y\),and the risk of the invoked\-tool majority is

ℛ​\(m\)=Pr⁡\(m≠y\)\.\\mathcal\{R\}\(m\)=\\Pr\(m\\neq y\)\.By definition, the override reward is

Roverride​\(p^θ,m,y\)=α​𝕀​\{p^θ=y,m≠y\}−α​𝕀​\{p^θ≠y,m=y\},R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)=\\alpha\\mathbb\{I\}\\\{\\hat\{p\}\_\{\\theta\}=y,\\ m\\neq y\\\}\-\\alpha\\mathbb\{I\}\\\{\\hat\{p\}\_\{\\theta\}\\neq y,\\ m=y\\\},whereα\>0\\alpha\>0is the override coefficient\. Taking expectation gives

𝔼​\[Roverride​\(p^θ,m,y\)\]\\displaystyle\\mathbb\{E\}\\left\[R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)\\right\]=α​Pr⁡\(p^θ=y,m≠y\)−α​Pr⁡\(p^θ≠y,m=y\)\.\\displaystyle=\\alpha\\Pr\(\\hat\{p\}\_\{\\theta\}=y,\\ m\\neq y\)\-\\alpha\\Pr\(\\hat\{p\}\_\{\\theta\}\\neq y,\\ m=y\)\.On the other hand,

ℛ​\(m\)−ℛ​\(πθ\)\\displaystyle\\mathcal\{R\}\(m\)\-\\mathcal\{R\}\(\\pi\_\{\\theta\}\)=Pr⁡\(m≠y\)−Pr⁡\(p^θ≠y\)\\displaystyle=\\Pr\(m\\neq y\)\-\\Pr\(\\hat\{p\}\_\{\\theta\}\\neq y\)=Pr⁡\(m≠y,p^θ=y\)\+Pr⁡\(m≠y,p^θ≠y\)\\displaystyle=\\Pr\(m\\neq y,\\hat\{p\}\_\{\\theta\}=y\)\+\\Pr\(m\\neq y,\\hat\{p\}\_\{\\theta\}\\neq y\)−Pr⁡\(p^θ≠y,m=y\)−Pr⁡\(p^θ≠y,m≠y\)\\displaystyle\\qquad\-\\Pr\(\\hat\{p\}\_\{\\theta\}\\neq y,m=y\)\-\\Pr\(\\hat\{p\}\_\{\\theta\}\\neq y,m\\neq y\)=Pr⁡\(m≠y,p^θ=y\)−Pr⁡\(m=y,p^θ≠y\)\.\\displaystyle=\\Pr\(m\\neq y,\\hat\{p\}\_\{\\theta\}=y\)\-\\Pr\(m=y,\\hat\{p\}\_\{\\theta\}\\neq y\)\.Therefore,

𝔼​\[Roverride​\(p^θ,m,y\)\]=α​\[ℛ​\(m\)−ℛ​\(πθ\)\]\.\\mathbb\{E\}\\left\[R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)\\right\]=\\alpha\\left\[\\mathcal\{R\}\(m\)\-\\mathcal\{R\}\(\\pi\_\{\\theta\}\)\\right\]\.Sinceα\>0\\alpha\>0, maximizing the expected override reward is equivalent to maximizing the binary risk improvement of the policy over the invoked\-tool majority\.

Now consider the improvement gap

𝒢​\(πθ\)=ℛsingle⋆−ℛ​\(πθ\)=ℛsingle⋆−ℛ​\(m\)\+1α​𝔼​\[Roverride​\(p^θ,m,y\)\]\.\\mathcal\{G\}\(\\pi\_\{\\theta\}\)=\\mathcal\{R\}^\{\\star\}\_\{\\mathrm\{single\}\}\-\\mathcal\{R\}\(\\pi\_\{\\theta\}\)=\\mathcal\{R\}^\{\\star\}\_\{\\mathrm\{single\}\}\-\\mathcal\{R\}\(m\)\+\\frac\{1\}\{\\alpha\}\\mathbb\{E\}\\left\[R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)\\right\]\.Becauseℛsingle⋆\\mathcal\{R\}^\{\\star\}\_\{\\mathrm\{single\}\}andℛ​\(m\)\\mathcal\{R\}\(m\)is independent ofθ\\theta, maximizing𝒢​\(πθ\)\\mathcal\{G\}\(\\pi\_\{\\theta\}\)is equivalent to maximizingRoverride​\(p^θ,m,y\)R\_\{\\mathrm\{override\}\}\(\\hat\{p\}\_\{\\theta\},m,y\)\. ∎

## Appendix BTool\-Calling Rollout Procedure

This appendix provides the general rollout template used by CSRL\. A rollout begins with a fixed system prompt that contains the task inputxx, the queryqq, and a<tools\>\.\.\.</tools\>block declaring the JSON schemas of theKKavailable tools\. The policyπθ\\pi\_\{\\theta\}then interacts with the tool\-augmented environment for up toTmaxT\_\{\\max\}assistant turns\.

At each assistant turn, the policy may either invoke one or more tools or terminate the rollout with a final answer\. A tool invocation follows the template:

<tool\_call\>​\{"name":Tk,"arguments":𝐚k\}​</tool\_call\>\.\\text\{\{\\color\[rgb\]\{0\.48046875,0\.12109375,0\.63671875\}\{<tool\\\_call\>\}\}\}\\\{\\texttt\{"name"\}:T\_\{k\},\\ \\texttt\{"arguments"\}:\\mathbf\{a\}\_\{k\}\\\}\\text\{\{\\color\[rgb\]\{0\.48046875,0\.12109375,0\.63671875\}\{</tool\\\_call\>\}\}\}\.The field"name"specifies the selected tool, and"arguments"contains the tool\-specific arguments constructed from the current input, query, and dialogue context\. Up toMMtool calls may be generated in a single assistant turn\. All tool calls generated in the same turn are executed in parallel by the environment, and their responses are inserted into the dialogue context before the next assistant turn\.

For each invoked toolTkT\_\{k\}, the environment returns its observation using the template:

<tool\_response\>​Tk​response:ok​</tool\_response\>,\\text\{\{\\color\[rgb\]\{0\.08203125,0\.39453125,0\.75390625\}\{<tool\\\_response\>\}\}\}\{\}\\,T\_\{k\}\\ \\text\{\{response\}\}:\\ o\_\{k\}\\,\\text\{\{\\color\[rgb\]\{0\.08203125,0\.39453125,0\.75390625\}\{</tool\\\_response\>\}\}\}\{\},whereoko\_\{k\}denotes the tool observation returned byTkT\_\{k\}\. The observation format is determined by the corresponding tool schema and task setting\. CSRL does not require all tools to share the same output format; the policy conditions on the returned observations through the dialogue context\.

The policy may terminate the rollout by generating a final answer:

<answer\>​p^θ​</answer\>,\\text\{\{\\color\[rgb\]\{0\.1796875,0\.48828125,0\.1953125\}\{<answer\>\}\}\}\{\}\\hat\{p\}\_\{\\theta\}\\text\{\{\\color\[rgb\]\{0\.1796875,0\.48828125,0\.1953125\}\{</answer\>\}\}\}\{\},wherep^θ∈𝒫\\hat\{p\}\_\{\\theta\}\\in\\mathcal\{P\}denotes the task\-specific prediction parsed from the terminal answer\. The rollout ends once a terminal answer is produced or the maximum number of assistant turns is reached\.

Tokens inserted by the environment, namely the<tool\_response\>blocks, are excluded from the policy\-gradient loss\. Only tokens autoregressively generated byπθ\\pi\_\{\\theta\}, including reasoning tokens, tool\-call tags, and the terminal answer, contribute to gradient updates\. This ensures that optimization is applied only to model\-generated decisions rather than to environment\-generated tool observations\.

## Appendix CBroader Impact

This work may improve the reliability of medical AI agents by learning how to combine complementary diagnostic tools on a case\-by\-case basis, potentially reducing missed findings and improving robustness across chest X\-ray datasets and clinical settings\. It can also help researchers better understand tool failure patterns and design safer medical decision\-support systems\. Potential risks include inheriting biases or label noise from pretrained models and datasets, and overreliance on automated outputs if used without clinical oversight\. Therefore, the system should be positioned as an assistive tool for clinicians rather than a replacement for medical professionals\.

## Appendix DTraining Details

#### Hyperparameters\.

Table[4](https://arxiv.org/html/2605.26691#A4.T4)reports the full GRPO training configuration\. Our implementation is based on VERL 0\.7\.1 with vLLM 0\.11\.0 as the rollout backend\.

Table 4:GRPO training hyperparameters\.
#### System Prompt\.

The following system prompt is prepended to every training and evaluation sample\.

> You are a radiology AI expert for chest X\-ray diagnosis\. You have access to 6 diagnostic tools with different strengths\. Each tool returns a score for the queried condition\. Query MULTIPLE tools and compare their results for a more reliable diagnosis\. Tools may disagree—use the disagreement to assess diagnostic uncertainty\. After consulting the tools, provide your final diagnosis probability \(0\.00\.0to1\.01\.0\) inside<answer\></answer\>tags\.

## Appendix ELLM Usage Statement

During the preparation of this manuscript, large language models were used only to polish the writing and provide suggestions on figure presentation\. The authors remain responsible for all technical content, experimental results, and scientific conclusions\.