Tag
This paper investigates whether hallucination in medical LLMs can be detected and controlled at the neuron level. The authors find that while hallucination signals are detectable across many neurons (AUROC 0.77-0.86), they are not easily corrected by steering those same neurons.
This paper presents a method that uses frozen medical large language model (LLM) representations as a shared embedding space to predict primary ICD diagnosis categories from both structured and unstructured electronic health record data, achieving improved accuracy over baseline methods on MIMIC-IV and showing transferability to MIMIC-III.
The author notes a surprising lack of publicly available APIs for medical-oriented LLMs, despite models like MedGemma and BioMistral existing on Hugging Face, and seeks information on any available options.
Introduces PhysAssistBench, a benchmark for evaluating LLMs in interactive doctor-patient-EHR assistance. Experiments show current models are unreliable in this setting, highlighting the need for coordinated capabilities.
This paper introduces AI-MASLD, a stress-audit framework for medical LLMs that reveals how benchmark accuracy can hide serious safety failures, and demonstrates that open-weight models can match or exceed proprietary ones on safety dimensions.
Introduces HiMed, a Hindi reasoning medical corpus and benchmark suite, and HiMed-8B, a Hindi-form medical reasoning LLM using decaying scaffolding reward, demonstrating improved Hindi medical reasoning and reduced English–Hindi accuracy gap.
Introduces OGCaReBench, a free-form retrieval benchmark for evaluating LLMs on clinical questions that require reasoning beyond standard guidelines. Experiments show that even the best model achieves only 56% accuracy, but retrieval augmentation boosts performance to 82%.
This paper presents a large-scale assessment of medical LLMs, including custom MedGPTs and open-source models, finding 25-30% exhibit low factual accuracy and 33.6-54.3% violate operational thresholds, highlighting systemic safety risks.
This article critiques Mark Kaplan's approach to fine-tuning medical LLMs via his platform healtthruth.ai, highlighting pitfalls in overriding foundational training for healthcare AI.