SAGEAgent: A Self-Evolving Agent for Cost-Aware Modality Acquisition in Multimodal Survival Prediction

# SAGEAgent: A Self-Evolving Agent for Cost-Aware Modality Acquisition in Multimodal Survival Prediction
Source: [https://arxiv.org/html/2607.09521](https://arxiv.org/html/2607.09521)
11institutetext:Vanderbilt University, Nashville TN 37235, USA22institutetext:Vanderbilt University Medical Center, Nashville TN, 37232 USA22email:yuankai\.huo@vanderbilt\.eduCan CuiZhengyi LuJunchao ZhuTianyuan YaoJunlin GuoJuming XiongYanfan ZhuYuechen YangBennett A\. LandmanYuankai Huo

###### Abstract

Does every cancer patient truly need a complete diagnostic workup for accurate survival prediction?In multimodal clinical oncology, diagnostic modalities follow a clinically mandated order of escalating burden—from demographics collected at intake to genomic profiling requiring specialized tissue analysis\. Current multimodal survival methods either assume all modalities are available or passively handle missing data, but none actively reason about whether acquiring the next modality is justified for a given patient along this ordered workflow\. We formulate this as a sequential decision problem and propose SAGEAgent \(SequentialAcquisitionGuided byExperience\), a self\-evolving LLM\-based clinical agent that decides which diagnostic modalities to acquire for each patient, balancing predictive accuracy against clinical invasiveness\. SAGEAgent reasons about each patient’s evolving diagnostic state through clinical tools that translate numerical predictions into text, an episodic memory that retrieves similar past cases, and a semantic memory that accumulates reusable decision patterns from experience\. Experiments on a glioma cohort combining TCGA\-LGG, TCGA\-GBM, and BraTS with four diagnostic modalities demonstrate that SAGEAgent achieves competitive survival prediction accuracy while reducing average acquisition burden by 55%\. Code is publicly available at:[https://github\.com/Chongyu1117/SAGEAgent](https://github.com/Chongyu1117/SAGEAgent)

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.09521v1/fig_intro.png)Figure 1:Overview of modality acquisition strategies\.\(a\) Existing approaches: static fusion applies all modalities uniformly, RL learns adaptive policies without explanation, and LLM\-based methods reason over independent tests without clinical ordering\. \(b\) SAGEAgent follows a clinically acquisition order, deciding at each stage whether to acquire or stop based on clinical tools, learned rules, and similar past cases\. Patient A stops early \(83% burden saved\); Patient B proceeds through all modalities\.Accurate survival prediction in oncology increasingly relies on integrating multiple diagnostic modalities\[[3](https://arxiv.org/html/2607.09521#bib.bib4),[15](https://arxiv.org/html/2607.09521#bib.bib3)\], ranging from routine demographics and radiology to invasive biopsy and molecular profiling\[[16](https://arxiv.org/html/2607.09521#bib.bib23)\]\. These modalities follow a strict clinical order: imaging precedes biopsy, which in turn provides the specimen for genomic analysis\. Previous multimodal fusion methods\[[5](https://arxiv.org/html/2607.09521#bib.bib48),[4](https://arxiv.org/html/2607.09521#bib.bib24),[8](https://arxiv.org/html/2607.09521#bib.bib54)\]combine all available modalities through a fixed strategy or handle missing data at test time, but treat prediction as a static problem—the model outputs a risk score from whatever is observed, without reasoning aboutwhether the next diagnostic step is worth it for this specific patient\.

Sequential test acquisition has been studied from two directions\. Reinforcement learning \(RL\) methods\[[2](https://arxiv.org/html/2607.09521#bib.bib22),[20](https://arxiv.org/html/2607.09521#bib.bib21)\]train cost\-aware acquisition policies but produce black\-box decisions that limit clinical trust\. Large language model \(LLM\) based approaches\[[13](https://arxiv.org/html/2607.09521#bib.bib20),[18](https://arxiv.org/html/2607.09521#bib.bib19),[1](https://arxiv.org/html/2607.09521#bib.bib18)\]offer transparent reasoning but operate on independent diagnostic tests without accounting for the sequential dependencies inherent in clinical modality acquisition\. Neither direction models the clinically mandated ordering \(see Fig\.[1](https://arxiv.org/html/2607.09521#S1.F1)\(a\) for a comparison\)\.

We propose SAGEAgent \(SequentialAcquisitionGuided byExperience\), a training\-free, self\-evolving LLM\-based framework for cost\-aware sequential modality acquisition in multimodal survival prediction \(Fig\.[1](https://arxiv.org/html/2607.09521#S1.F1)\(b\)\)\. At each stage of the clinical order \(demographics→\\rightarrowradiology→\\rightarrowpathology→\\rightarrowgenomics\), SAGEAgent decides whether to acquire or stop using three signal sources: clinical tools that translate numerical predictions into text, an episodic memory that retrieves similar past cases, and a semantic memory that accumulates decision rules through periodic self\-reflection—all without gradient updates\. Evaluated on a glioma cohort combining TCGA\-LGG\[[19](https://arxiv.org/html/2607.09521#bib.bib16)\], TCGA\-GBM\[[21](https://arxiv.org/html/2607.09521#bib.bib17)\], and BraTS\[[17](https://arxiv.org/html/2607.09521#bib.bib15)\]with nested 5×\\times5 cross\-validation, SAGEAgent achieves competitive survival prediction accuracy while reducing acquisition burden by 55% compared to full workup, providing transparent reasoning for each patient’s acquisition pathway\.

Our contributions are threefold:

1. 1\.At the learning framework level, we propose SAGEAgent, a novel framework that formulates cost\-aware modality acquisition for multimodal survival prediction as a sequential decision\-making problem, while explicitly respecting the clinical ordering of diagnostic procedures\.
2. 2\.At the training strategy level, we introduce a dual\-memory self\-evolution mechanism that autonomously discovers decision rules from experience, eliminating the need for gradient\-based updates or hand\-crafted priors\.
3. 3\.At the interpretability level, we provide fully auditable reasoning pathways in SAGEAgent, where each acquisition decision is grounded in explicit chain\-of\-thought reasoning, thereby bridging cost\-aware modality acquisition with clinical transparency\.

![Refer to caption](https://arxiv.org/html/2607.09521v1/fig_method.png)Figure 2:SAGEAgent architecture and reasoning\.SAGEAgent Flow: At each stage, the frozen predictor produces the patient embedding, risk score, and calibrated uncertainty\. Three signal sources feed the frozen LLM: clinical tools convert numerical outputs to text, episodic memory retrieves similar patients and past decisions, and semantic memory provides learned decision rules\.Chain\-of\-Thought Reasoning: A real reasoning trace after radiology, with highlights tracing each step to its source component\. The agent synthesizes all three signals and decides to stop, skipping invasive pathology\.Self\-Evolution: Episodes accumulate with outcome labels; the reflection module analyzes which rules led to good or poor outcomes and updates semantic memory accordingly, forming a closed loop without gradient updates\.
## 2Method

### 2\.1Problem Formulation

Consider a patient withNNdiagnostic modalities\{m1,…,mN\}\\\{m\_\{1\},\\ldots,m\_\{N\}\\\}arranged in a clinically mandated acquisition orderm1→m2→⋯→mNm\_\{1\}\\rightarrow m\_\{2\}\\rightarrow\\cdots\\rightarrow m\_\{N\}\. For glioma, this order is demographics→\\rightarrowradiology→\\rightarrowpathology→\\rightarrowgenomics, reflecting real clinical dependencies\[[16](https://arxiv.org/html/2607.09521#bib.bib23)\]\. Each modalitymim\_\{i\}carries a clinical burdenb​\(mi\)∈\[0,1\]b\(m\_\{i\}\)\\in\[0,1\], quantified via multi\-criteria decision analysis \(MCDA\)\[[11](https://arxiv.org/html/2607.09521#bib.bib7)\]over monetary cost, turnaround time, patient invasiveness, and infrastructure requirements \(details in Sec\.[3\.1](https://arxiv.org/html/2607.09521#S3.SS1)\)\. At decision staget∈\{1,…,N−1\}t\\in\\\{1,\\ldots,N\{\-\}1\\\}, the agent has already acquired the firstttmodalities\. The decision is binary: acquiremt\+1m\_\{t\+1\}\(incurring burdenb​\(mt\+1\)b\(m\_\{t\+1\}\)\), or stop and predict\. The goal is to maximize prediction accuracy while minimizing total acquisition burdenBt=∑i=1tb​\(mi\)B\_\{t\}=\\sum\_\{i=1\}^\{t\}b\(m\_\{i\}\)\. Fig\.[2](https://arxiv.org/html/2607.09521#S1.F2)provides an overview of the proposed framework\.

### 2\.2Multimodal Predictor with Calibrated Uncertainty

Survival predictor\.The prediction backbone is a Transformer\-based multimodal encoder that handles missing modalities via learnable mask tokens\[[9](https://arxiv.org/html/2607.09521#bib.bib14),[8](https://arxiv.org/html/2607.09521#bib.bib54)\]\. It maps each patient’s available modalities to a fused embedding𝐞t∈ℝd\\mathbf\{e\}\_\{t\}\\in\\mathbb\{R\}^\{d\}, from which a Cox proportional hazards head\[[7](https://arxiv.org/html/2607.09521#bib.bib13)\]produces a scalar risk scorertr\_\{t\}\. The predictor is trained with modality dropout andfrozen during agent operation\. Architecture and training details are in Sec\.[3\.2](https://arxiv.org/html/2607.09521#S3.SS2)\.

Calibrated uncertainty head\.On top of the frozen predictor, we train a lightweight uncertainty head that estimates whether the current prediction would change if more modalities were acquired\. Given the embedding𝐞t\\mathbf\{e\}\_\{t\}and acquisition mask𝐚t\\mathbf\{a\}\_\{t\}, it outputs a calibrated probability:

ut=σ​\(fϕ​\(\[𝐞t,𝐚t\]\)\)u\_\{t\}=\\sigma\\\!\\bigl\(f\_\{\\phi\}\(\[\\mathbf\{e\}\_\{t\},\\,\\mathbf\{a\}\_\{t\}\]\)\\bigr\)\(1\)wherefϕf\_\{\\phi\}is a lightweight MLP andσ\\sigmais the sigmoid function\. The training label is 1 when the absolute difference between the current and full\-modality risk scores exceeds a thresholdτ\\tau, and 0 otherwise\. Training data is generated by enumerating clinical\-order prefix masks on complete\-modality patients\. The resultingut∈\[0,1\]u\_\{t\}\\in\[0,1\]is patient\-specific and mask\-specific, giving the agent a calibrated signal of whether additional data is likely to matter\.

### 2\.3SAGEAgent: LLM\-Based Acquisition Agent

SAGEAgent is built on a frozen LLM that receives no gradient updates\. At each stage, it gathers diagnostic signals, retrieves relevant experience, and outputs a binary decision \(acquire or predict\) with a chain\-of\-thought explanation\. Its behavior improves solely through two memory systems following the episodic–semantic distinction in cognitive science\[[24](https://arxiv.org/html/2607.09521#bib.bib12)\]\.

Clinical decision tools\.Two tools translate the frozen predictor’s numerical outputs into text\. TheUncertainty Toolreportsutu\_\{t\}from Eq\.[1](https://arxiv.org/html/2607.09521#S2.E1)with a categorical level \(very low to very high\) based on quintile thresholds from the training distribution\. TheSurvival Predictor Toolreports the risk scorertr\_\{t\}and its position relative to the training cohort \(below/around/above average\)\. Both tools present outputs as natural\-language sentences so that all reasoning occurs in text space\.

Episodic memory\.The episodic memory provides case\-based reasoning through two complementary retrieval mechanisms\.Case retrievaldraws from a FAISS\-indexed\[[10](https://arxiv.org/html/2607.09521#bib.bib8)\]database of training patients, returning thekknearest neighbors by cosine similarity onℓ2\\ell\_\{2\}\-normalized embeddings together with their event status \(survived or deceased\) and survival time\. This provides a static clinical reference: which training patients resemble the current one, and how long did they survive\.Episode recalldraws from the agent’s own past decisions\. When the agent acquiresmt\+1m\_\{t\+1\}, it receives a stage rewardRstage=−b​\(mt\+1\)\+max⁡\(0,ut−ut\+1\)R\_\{\\text\{stage\}\}=\-b\(m\_\{t\+1\}\)\+\\max\(0,\\,u\_\{t\}\-u\_\{t\+1\}\)that penalizes cost and rewards uncertainty reduction\. When the agent stops at stagett, a terminal rewardRterm=\(1−ut\)−BtR\_\{\\text\{term\}\}=\(1\-u\_\{t\}\)\-B\_\{t\}favors early stopping at low uncertainty\. Retrieval is stage\-matched and re\-ranked by the sum of cosine similaritysim​\(𝐞t,𝐞j\)\\text\{sim\}\(\\mathbf\{e\}\_\{t\},\\mathbf\{e\}\_\{j\}\)and min\-max normalized episode rewardr~j∈\[0,1\]\\tilde\{r\}\_\{j\}\\in\[0,1\], returning the top\-kkepisodes, including the action taken and its outcome\.

Semantic memory and self\-evolution\.The semantic memory stores interpretable decision rules discovered through periodic self\-reflection\. Each rule is indexed by acquisition stagettand action directiond∈\{stop,acquire\}d\\in\\\{\\text\{stop\},\\text\{acquire\}\\\}\. After each episode, we record whether the agent followed or violated each relevant rule\. EveryNrN\_\{r\}episodes \(one reflection cycle\), episodes are ranked by total reward, and those in the top and bottom quartiles are labeled good and poor respectively\. Rule effectiveness is the fraction of followed episodes that led to good outcomes\. The reflection module receives the effectiveness report and proposes new rules, which are verified for direction clarity and novelty\. Rules that consistently lead to poor outcomes are deprecated, forming a closed loop where rules are discovered, validated, and either reinforced or removed\.

## 3Experiments & Results

### 3\.1Dataset and Evaluation Protocol

Dataset and split\.We evaluate on a glioma cohort of 962 patients \(803 TCGA\-LGG/GBM\[[19](https://arxiv.org/html/2607.09521#bib.bib16),[21](https://arxiv.org/html/2607.09521#bib.bib17)\]and 159 BraTS\[[17](https://arxiv.org/html/2607.09521#bib.bib15)\]\) spanning four diagnostic modalities in clinical order: demographics \(Demo\.\), radiology \(Rad\.\), pathology \(Path\.\), and genomics \(Gen\.\), each encoded into a 32\-dimensional feature vector by pretrained extractors\[[8](https://arxiv.org/html/2607.09521#bib.bib54)\]\. Of the 962 patients, 170 possess all four modalities and form the evaluation set\. We adopt nested5×55\{\\times\}5cross\-validation\. The outer loop splits 170 patients into 5 folds \(136/34\) for agent training and evaluation\. Each inner 5\-fold trains the predictor on the 136 patients plus 792 partial\-modality patients as augmentation via modality dropout, yielding 25 independent pipelines\. Per\-patient decisions are aggregated by majority vote \(≥\\geq3/5 inner models\)\. We report mean±\\pmstd across 5 outer folds\.

Clinical burden\.We quantify modality cost via MCDA\[[11](https://arxiv.org/html/2607.09521#bib.bib7)\]over monetary cost, turnaround time, invasiveness, and infrastructure requirements\. After normalization, per\-modality burdens are 0\.03 \(demographics\), 0\.14 \(radiology\), 0\.53 \(pathology\), and 0\.30 \(genomics\), totalingB=1\.0B\{=\}1\.0for a full workup\.

Metrics\.We report C\-index\[[14](https://arxiv.org/html/2607.09521#bib.bib10)\], average burden, and hypervolumeHV=\(c−0\.5\)×\(1\.0−b\)\\text\{HV\}=\(c\-0\.5\)\\times\(1\.0\-b\)which captures the accuracy–cost trade\-off as a single scalar\.

Table 1:Core results on the glioma survival cohort \(170 complete\-modality patients\)\.Top \(a\) C\-index in ascending order\. RL\-based methods form a lower cluster around 0\.73, while the remaining six methods achieve C\-index above 0\.8\. Top \(b\) C\-index \(solid\) and burden \(hatched\) for the six methods with C\-index\>\>0\.8\. Bottom: per\-method breakdown\.= 170 patients,= 5 patients, with shades varying linearly between these values\.†\\dagger= predictor backbone used by SAGEAgent\.![[Uncaptioned image]](https://arxiv.org/html/2607.09521v1/fig_results.png)

CategoryMethodC\-index↑\\uparrowBurden↓\\downarrow\(%saved\)Demo\.Rad\.Path\.Gen\.StaticConcat\. Fusion0\.812±\\pm\.0401\.000 \(0%\)FusionMCAT\[[6](https://arxiv.org/html/2607.09521#bib.bib6)\]0\.817±\\pm\.0461\.000 \(0%\)MMD\[[8](https://arxiv.org/html/2607.09521#bib.bib54)\]0\.822±\\pm\.0321\.000 \(0%\)Self\-Att\. Fusion†0\.825±\\pm\.0501\.000 \(0%\)RL\-basedBernardino et al\.\[[2](https://arxiv.org/html/2607.09521#bib.bib22)\]0\.742±\\pm\.0580\.125 \(88%\)AcquisitionPRECISE\-AS\[[20](https://arxiv.org/html/2607.09521#bib.bib21)\]0\.743±\\pm\.0590\.127 \(87%\)PPO\[[22](https://arxiv.org/html/2607.09521#bib.bib31)\]0\.729±\\pm\.0700\.124 \(88%\)LLM\-basedReflexion\[[23](https://arxiv.org/html/2607.09521#bib.bib5)\]0\.823±\\pm\.0490\.872 \(13%\)AcquisitionSAGEAgent0\.813±\\pm\.0460\.451 \(55%\)

### 3\.2Implementation Details

Predictor\.The backbone is a 2\-layer Transformer encoder \(d=128d\{=\}128, 4 heads\) with per\-modality MLP projections fromℝ32\\mathbb\{R\}^\{32\}toℝ128\\mathbb\{R\}^\{128\}and learnable mask tokens for missing modalities, trained with Cox partial likelihood, reconstruction, and alignment losses under 50% modality dropout \(∼\{\\sim\}458K parameters\)\. The calibrated uncertainty head \(Eq\.[1](https://arxiv.org/html/2607.09521#S2.E1)\) adds∼\{\\sim\}6\.5K parameters trained with BCE loss, withτ=0\.3\\tau\{=\}0\.3set by calibration quality \(AUROC = 0\.908, ECE = 0\.110\) on inner\-fold validation after post\-hoc temperature scaling\[[12](https://arxiv.org/html/2607.09521#bib.bib11)\]\.

SAGEAgentuses Qwen\-2\.5\-7B\-Instruct\[[25](https://arxiv.org/html/2607.09521#bib.bib9)\]with fully frozen weights, running on a single NVIDIA A6000 GPU\. During experience accumulation, each training patient is processed for 3 episodes, yielding∼\{\\sim\}408 episodes per fold\. Episodic memory retrievesk=3k\{=\}3stage\-matched neighbors via FAISS\[[10](https://arxiv.org/html/2607.09521#bib.bib8)\]\. Semantic memory maintains up to 10 active rules with reflection triggered every 10 patients\.

### 3\.3Core Results

Table[1](https://arxiv.org/html/2607.09521#S3.T1)and compare SAGEAgent against static fusion baselines and acquisition methods under the nested 5×\\times5 CV protocol described in Sec\.[3\.1](https://arxiv.org/html/2607.09521#S3.SS1)\.

RL methods reduce cost at the expense of accuracy\.A C\-index gap of approximately 0\.07 separates the nine methods into two clusters\.\(Table[1](https://arxiv.org/html/2607.09521#S3.T1)top \(a\)\) All three RL baselines reduce burden to∼\{\\sim\}0\.12 but drop C\-index to∼\{\\sim\}0\.73, effectively discarding informative modalities\. Static fusion methods require full workup by design, and Reflexion reduces burden by only 13%\.

SAGEAgent maintains accuracy while reducing burden by 55%\.As shown in Table[1](https://arxiv.org/html/2607.09521#S3.T1)top \(b\), SAGEAgent achieves a C\-index of 0\.813, within 0\.012 of the full\-modality backbone, while reducing burden to 0\.451\. Compared to Reflexion, SAGEAgent achieves4\.2×4\.2\{\\times\}greater burden reduction with a C\-index difference of only 0\.010\. Among compared methods, SAGEAgent is the only one that achieves meaningful burden reduction without compromising accuracy\.

Table 2:Component ablation\.Each row adds one component to the Base LLM\. All configurations use frozen Qwen\-2\.5\-7B\-Instruct with majority\-vote aggregation\. Bold indicates best result\.= 170 patients,= 5 patients, with shades varying linearly\.
### 3\.4Ablation Study

Table[2](https://arxiv.org/html/2607.09521#S3.T2)isolates the contribution of each component by progressively adding tools, episodic memory, and semantic memory to the frozen LLM\. Adding tools alone raises burden from 0\.798 to 0\.843 because without past experience to contextualize uncertainty and risk signals, the LLM defaults to acquiring more\. Episodic memory provides the largest individual burden reduction \(0\.843→\\to0\.611\) by surfacing cases where early stopping succeeded, while semantic memory offers a complementary effect \(0\.843→\\to0\.706\) through interpretable stopping criteria\. Combined, SAGEAgent achieves the lowest burden \(0\.451\) and highest HV \(0\.172\), a2\.6×2\.6\{\\times\}improvement over the Base LLM\.

Table 3:Rules discovered by semantic memory across five folds\.Eff\. = fraction of rule\-following episodes with top\-quartile reward \(random baseline = 0\.25\)\.nn= total episodes where the rule was followed across folds\. Stability indicates whether the rule, once discovered, was ever deprecated and re\-proposed during training\.
### 3\.5Self\-Evolution Analysis

The ablation in Sec\.[3\.4](https://arxiv.org/html/2607.09521#S3.SS4)shows that semantic memory reduces burden, but does not reveal what rules it learns\. We now examine whether the discovered rules are consistent across folds\. Each outer fold runs 10 reflection cycles from an empty rule set with no shared initialization\.

Semantic memory discovers consistent rules across folds\.Table[3](https://arxiv.org/html/2607.09521#S3.T3)lists the four rules emerging in at least four of five folds, all achieving effectiveness well above the 0\.25 random baseline and collectively covering over 2,100 decisions\. Each fold generates 10\.8 candidate rules on average, of which 4\.6 survive pruning\.

Stopping rules are easier to learn than acquisition rules\.The two stopping rules emerge in the first two reflection cycles and are never deprecated\. In contrast, the acquisition rule “after radiology, acquire pathology” accumulates 13 deprecated instances across folds before a stable version survives in four of five\. Reflection logs reveals a two\-phase pattern: cycles 1–2 establish core stopping rules that persist unchanged, while cycles 3–10 are dominated by acquire\-rule exploration where most proposals are deprecated due to near\-zero effectiveness\.

## 4Conclusion

We presented SAGEAgent, a training\-free LLM\-based framework for cost\-aware sequential modality acquisition in multimodal survival prediction\. On a glioma cohort, SAGEAgent reduces acquisition burden by 55% while maintaining accuracy within 0\.012 C\-index of full\-modality fusion, with each decision grounded in auditable reasoning\. Notably, the agent learns to skip invasive pathology for half of the cohort, suggesting that a substantial fraction of glioma patients may not require biopsy for reliable prognostication\. Self\-evolution analysis reveals that stopping rules converge reliably while acquisition rules are inherently harder to learn\. Future work includes evaluation on larger multi\-site cohorts, other cancer types, and prospective clinical validation\.

## References

- \[1\]D\. Bani\-Harouni, C\. Pellegrini, E\. Özsoy, M\. Keicher, and N\. Navab\(2025\)Language agents for hypothesis\-driven clinical decision making with reinforcement learning\.arXiv preprint arXiv:2506\.13474\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p2.1)\.
- \[2\]G\. Bernardino, A\. Jonsson, F\. Loncaric, P\. Martí Castellote, M\. Sitges, P\. Clarysse, and N\. Duchateau\(2022\)Reinforcement learning for active modality selection during diagnosis\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 592–601\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p2.1),[Table 1](https://arxiv.org/html/2607.09521#S3.T1.12.8.3.1.1)\.
- \[3\]K\. M\. Boehm, P\. Khosravi, R\. Vanguri, J\. Gao, and S\. P\. Shah\(2022\)Harnessing multimodal data integration to advance precision oncology\.Nature Reviews Cancer22\(2\),pp\. 114–126\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p1.1)\.
- \[4\]N\. Braman, J\. W\. Gordon, E\. T\. Goossens, C\. Willis, M\. C\. Stumpe, and J\. Venkataraman\(2021\)Deep orthogonal fusion: multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data\.InInternational conference on medical image computing and computer\-assisted intervention,pp\. 667–677\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p1.1)\.
- \[5\]R\. J\. Chen, M\. Y\. Lu, J\. Wang, D\. F\. Williamson, S\. J\. Rodig, N\. I\. Lindeman, and F\. Mahmood\(2020\)Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis\.IEEE Transactions on Medical Imaging41\(4\),pp\. 757–770\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p1.1)\.
- \[6\]R\. J\. Chen, M\. Y\. Lu, W\. Weng, T\. Y\. Chen, D\. F\. Williamson, T\. Manz, M\. Shady, and F\. Mahmood\(2021\)Multimodal co\-attention transformer for survival prediction in gigapixel whole slide images\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 4015–4025\.Cited by:[Table 1](https://arxiv.org/html/2607.09521#S3.T1.8.4.3.1.1)\.
- \[7\]D\. R\. Cox\(1972\)Regression models and life\-tables\.Journal of the royal statistical society: Series B \(methodological\)34\(2\),pp\. 187–202\.Cited by:[§2\.2](https://arxiv.org/html/2607.09521#S2.SS2.p1.2)\.
- \[8\]C\. Cui, H\. Liu, Q\. Liu, R\. Deng, Z\. Asad, Y\. Wang, S\. Zhao, H\. Yang, B\. A\. Landman, and Y\. Huo\(2022\)Survival prediction of brain cancer with incomplete radiology, pathology, genomic, and demographic data\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 626–635\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.09521#S2.SS2.p1.2),[§3\.1](https://arxiv.org/html/2607.09521#S3.SS1.p1.3),[Table 1](https://arxiv.org/html/2607.09521#S3.T1.9.5.3.1.1)\.
- \[9\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§2\.2](https://arxiv.org/html/2607.09521#S2.SS2.p1.2)\.
- \[10\]M\. Douze, A\. Guzhva, C\. Deng, J\. Johnson, G\. Szilvasy, P\. Mazaré, M\. Lomeli, L\. Hosseini, and H\. Jégou\(2025\)The faiss library\.IEEE Transactions on Big Data\.Cited by:[§2\.3](https://arxiv.org/html/2607.09521#S2.SS3.p3.9),[§3\.2](https://arxiv.org/html/2607.09521#S3.SS2.p2.2)\.
- \[11\]A\. Guitouni and J\. Martel\(1998\)Tentative guidelines to help choosing an appropriate mcda method\.European journal of operational research109\(2\),pp\. 501–521\.Cited by:[§2\.1](https://arxiv.org/html/2607.09521#S2.SS1.p1.13),[§3\.1](https://arxiv.org/html/2607.09521#S3.SS1.p2.1)\.
- \[12\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§3\.2](https://arxiv.org/html/2607.09521#S3.SS2.p1.6)\.
- \[13\]P\. Hager, F\. Jungmann, R\. Holland, K\. Bhagat, I\. Hubrecht, M\. Knauer, J\. Vielhauer, M\. Makowski, R\. Braren, G\. Kaissis,et al\.\(2024\)Evaluation and mitigation of the limitations of large language models in clinical decision\-making\.Nature medicine30\(9\),pp\. 2613–2622\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p2.1)\.
- \[14\]F\. E\. Harrell, R\. M\. Califf, D\. B\. Pryor, K\. L\. Lee, and R\. A\. Rosati\(1982\)Evaluating the yield of medical tests\.Jama247\(18\),pp\. 2543–2546\.Cited by:[§3\.1](https://arxiv.org/html/2607.09521#S3.SS1.p3.1)\.
- \[15\]J\. Lipkova, R\. J\. Chen, B\. Chen, M\. Y\. Lu, M\. Barbieri, D\. Shao, A\. J\. Vaidya, C\. Chen, L\. Zhuang, D\. F\. Williamson,et al\.\(2022\)Artificial intelligence for multimodal data integration in oncology\.Cancer cell40\(10\),pp\. 1095–1110\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p1.1)\.
- \[16\]D\. N\. Louis, A\. Perry, P\. Wesseling, D\. J\. Brat, I\. A\. Cree, D\. Figarella\-Branger, C\. Hawkins, H\. Ng, S\. M\. Pfister, G\. Reifenberger,et al\.\(2021\)The 2021 who classification of tumors of the central nervous system: a summary\.Neuro\-oncology23\(8\),pp\. 1231–1251\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.09521#S2.SS1.p1.13)\.
- \[17\]B\. H\. Menze, A\. Jakab, S\. Bauer, J\. Kalpathy\-Cramer, K\. Farahani, J\. Kirby, Y\. Burren, N\. Porz, J\. Slotboom, R\. Wiest,et al\.\(2014\)The multimodal brain tumor image segmentation benchmark \(brats\)\.IEEE transactions on medical imaging34\(10\),pp\. 1993–2024\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p3.4),[§3\.1](https://arxiv.org/html/2607.09521#S3.SS1.p1.3)\.
- \[18\]H\. Nori, M\. Daswani, C\. Kelly, S\. Lundberg, M\. T\. Ribeiro, M\. Wilson, X\. Liu, V\. Sounderajah, J\. Carlson, M\. P\. Lungren,et al\.\(2025\)Sequential diagnosis with language models\.arXiv preprint arXiv:2506\.22405\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p2.1)\.
- \[19\]N\. Pedano, A\. E\. Flanders, L\. Scarpace, T\. Mikkelsen, J\. M\. Eschbacher, B\. Hermes, and Q\. Ostrom\(2016\)Radiology data from the cancer genome atlas low grade glioma \[tcga\-lgg\] collection\.The cancer imaging archive2\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p3.4),[§3\.1](https://arxiv.org/html/2607.09521#S3.SS1.p1.3)\.
- \[20\]A\. Saadat, N\. Hashemi, H\. Vaseli, M\. Y\. Tsang, C\. Luong, M\. Van de Panne, T\. S\. Tsang, and P\. Abolmaesumi\(2025\)PRECISE\-as: personalized reinforcement learning for efficient point\-of\-care echocardiography in aortic stenosis diagnosis\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 474–484\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p2.1),[Table 1](https://arxiv.org/html/2607.09521#S3.T1.13.9.3.1.1)\.
- \[21\]L\. Scarpace, T\. Mikkelsen, S\. Cha, S\. Rao, S\. Tekchandani, D\. Gutman, J\. H\. Saltz, B\. J\. Erickson, N\. Pedano, A\. E\. Flanders,et al\.\(2016\)The cancer genome atlas glioblastoma multiforme collection \(tcga\-gbm\)\.The Cancer Imaging Archive\.Cited by:[§1](https://arxiv.org/html/2607.09521#S1.p3.4),[§3\.1](https://arxiv.org/html/2607.09521#S3.SS1.p1.3)\.
- \[22\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Table 1](https://arxiv.org/html/2607.09521#S3.T1.14.10.3.1.1)\.
- \[23\]N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. Yao\(2023\)Reflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[Table 1](https://arxiv.org/html/2607.09521#S3.T1.15.11.3.1.1)\.
- \[24\]E\. Tulvinget al\.\(1972\)Episodic and semantic memory\.Organization of memory1\(381\-403\),pp\. 1\.Cited by:[§2\.3](https://arxiv.org/html/2607.09521#S2.SS3.p1.1)\.
- \[25\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.2](https://arxiv.org/html/2607.09521#S3.SS2.p2.2)\.