Tag
LangSmith is helping healthcare organizations transform expert reviews into reusable evaluations for AI agents, improving patient care workflows.
This paper presents InterviewPlayground, a clinician-grounded evaluation platform for AI-assisted psychiatric intake using simulated patients, showing that an LLM interviewer recovers more clinical items but makes more unsupported inferences compared to clinicians.
The study evaluates 16 large language models against 60 practicing TCM physicians using real-world clinical cases, finding that LLMs achieve higher expert scores in some areas but show discrepancies and safety concerns. It highlights the potential and limitations of LLMs in TCM decision support.
This paper introduces Japanese Stroke LLM Evaluation, a conversational benchmark for assessing large language models in safe Japanese stroke care, showing that Claude Fable 5 achieved the highest score and met safety thresholds while other models made critical mistakes.
The paper investigates using Large Language Models as post-hoc auditors to evaluate symbolic regression models for interpretability and medical plausibility, with clinician assessments showing comparative model rankings are more favorably perceived than term-level interpretations.
This article introduces oncothresh, a Python library and web dashboard for evaluating oncology AI models at specific clinical decision thresholds, providing metrics like sensitivity, specificity, and decision-curve analysis to fill gaps in existing benchmarks.
MedBench v5 is a dynamic, process-oriented benchmark for clinical multimodal models that integrates hallucination detection and stress testing, moving beyond static QA to evaluate reasoning and stability under information-flow stressors.
This paper presents a structured framework for benchmarking generative, multimodal, and agentic AI in healthcare, addressing the gap between high benchmark scores and real-world clinical reliability, safety, and relevance.
OpenAI assembled a team of practicing doctors to evaluate and improve ChatGPT's health-related responses using real clinical experience, aiming to enhance accuracy and communication methods, ultimately democratizing medical knowledge.