Surrogate modeling for interpreting black-box LLMs in medical predictions
Summary
Researchers propose a surrogate modeling framework to quantify and interpret latent medical knowledge encoded in black-box LLMs, revealing both valid associations and persistent racial biases.
View Cached Full Text
Cached at: 04/23/26, 10:03 AM
# Surrogate modeling for interpreting black-box LLMs in medical predictions Source: [https://arxiv.org/abs/2604.20331](https://arxiv.org/abs/2604.20331) Authors:[Changho Han](https://arxiv.org/search/cs?searchtype=author&query=Han,+C)\(1\),[Songsoo Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+S)\(2\),[Dong Won Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+D+W)\(2\),[Leo Anthony Celi](https://arxiv.org/search/cs?searchtype=author&query=Celi,+L+A)\(3, 4 and 5\),[Jaewoong Kim](https://arxiv.org/search/cs?searchtype=author&query=Kim,+J)\(2\),[SungA Bae](https://arxiv.org/search/cs?searchtype=author&query=Bae,+S)\(6 and 7\),[Dukyong Yoon](https://arxiv.org/search/cs?searchtype=author&query=Yoon,+D)\(2, 7 and 8\) \(\(1\) Medical Big Data Research Center, Seoul National University Medical Research Center, Seoul National University College of Medicine, Seoul, Republic of Korea, \(2\) Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, Republic of Korea, \(3\) Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA, \(4\) Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA, \(5\) Department of Biostatistics, Harvard T\.H\. Chan School of Public Health, Boston, MA, USA, \(6\) Department of Cardiology, Yongin Severance Hospital, Yonsei University College of Medicine, Yongin, Republic of Korea, \(7\) Center for Digital Health, Yongin Severance Hospital, Yonsei University Health System, Yongin, Republic of Korea, \(8\) Institute for Innovation in Digital Healthcare, Severance Hospital, Seoul, Republic of Korea\) [View PDF](https://arxiv.org/pdf/2604.20331) > Abstract:Large language models \(LLMs\), trained on vast datasets, encode extensive real\-world knowledge within their parameters, yet their black\-box nature obscures the mechanisms and extent of this encoding\. Surrogate modeling, which uses simplified models to approximate complex systems, can offer a path toward better interpretability of black\-box models\. We propose a surrogate modeling framework that quantitatively explains LLM\-encoded knowledge\. For a specific hypothesis derived from domain knowledge, this framework approximates the latent LLM knowledge space using observable elements \(input\-output pairs\) through extensive prompting across a comprehensive range of simulated scenarios\. Through proof\-of\-concept experiments in medical predictions, we demonstrate our framework's effectiveness in revealing the extent to which LLMs "perceive" each input variable in relation to the output\. Particularly, given concerns that LLMs may perpetuate inaccuracies and societal biases embedded in their training data, our experiments using this framework quantitatively revealed both associations that contradict established medical knowledge and the persistence of scientifically refuted racial assumptions within LLM\-encoded knowledge\. By disclosing these issues, our framework can act as a red\-flag indicator to support the safe and reliable application of these models\. ## Submission history From: Songsoo Kim MD PhD \[[view email](https://arxiv.org/show-email/fa8d5092/2604.20331)\] **\[v1\]**Wed, 22 Apr 2026 08:26:23 UTC \(2,194 KB\)
Similar Articles
LLMs for Cardiovascular Risk Prediction from Structured Clinical Data
This paper presents a hybrid framework that combines structured clinical data with LLM-generated narratives for coronary artery disease prediction, achieving high fidelity in variable extraction and comparing ML models with LLM-based zero-shot and few-shot classification.
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
This survey examines recent progress in medical LLMs, presenting a dual-view approach that connects clinical practice with computational methods, and introduces a benchmark dataset for evaluating medical reasoning capabilities across 18 state-of-the-art models.
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
This paper introduces AI-MASLD, a stress-audit framework for medical LLMs that reveals how benchmark accuracy can hide serious safety failures, and demonstrates that open-weight models can match or exceed proprietary ones on safety dimensions.
TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs
TRIAGE is a framework that trains LLMs to generate dialectical reasoning for continuous risk scoring from irregularly sampled medical time series, achieving improved calibration and interpretability.
Large Language Models as Unified Multimodal Learners for Clinical Prediction
The paper proposes converting multimodal patient data (text, labs, vitals) into a single natural language sequence and fine-tuning LLMs for clinical prediction, achieving comparable or better performance than specialized fusion architectures across three tasks.