When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening
Summary
This paper introduces a SCID-anchored benchmark of 555 interviews to evaluate five LLMs for psychiatric screening, finding that while models show potential, they tend to discount symptom evidence in the presence of preserved functioning or protective context, requiring careful validation.
View Cached Full Text
Cached at: 05/25/26, 08:59 AM
# When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening Source: [https://arxiv.org/abs/2605.23148](https://arxiv.org/abs/2605.23148) [View PDF](https://arxiv.org/pdf/2605.23148) > Abstract:As demand for mental health care outpaces clinician\-delivered assessment, scalable screening tools are increasingly needed\. Large language models \(LLMs\) may identify psychiatric risk from patient narratives, but their reliability across diagnoses, demographic subgroups, and evidence\-use patterns remains uncertain\. We introduce a SCID\-anchored benchmark of 555 semi\-structured experiential interviews paired with diagnostic reference labels for anxiety disorder, major depressive disorder, post\-traumatic stress disorder, and any current mental health disorder\. Using zero\-shot task\-specific prompting, we evaluated five state\-of\-the\-art LLMs and examined whether false\-negative errors reflected missed psychiatric evidence or differential weighting of symptom, functional\-impairment, and protective\-context cues\. Performance varied across tasks and models, with accuracy ranging from 0\.49 to 0\.86 and Matthews correlation coefficients from 0\.16 to 0\.38\. GPT\-4\.1 Mini and GPT\-5 Mini showed the most consistent disorder\-specific accuracy\. Subgroup analyses found higher depression\-classification accuracy among male than female participants, no consistent age\-related pattern, and modest non\-uniform variation across race strata\. Evidence\-integration analyses showed that false\-negative anxiety and PTSD classifications often contained explicit symptom evidence but were accompanied by preserved functioning, coping ability, or social support\. Functional\-impairment evidence shifted model outputs toward positive classifications, whereas protective\-context evidence shifted outputs away\. These findings suggest that LLMs may support scalable psychiatric screening, but their tendency to discount symptom evidence in the presence of preserved functioning or protective context requires careful validation before clinical deployment\. ## Submission history From: Jianfeng Zhu \[[view email](https://arxiv.org/show-email/c8efb6c7/2605.23148)\] **\[v1\]**Fri, 22 May 2026 01:53:03 UTC \(1,302 KB\)
Similar Articles
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Researchers introduce MedSP1000, a 1,638-case interactive benchmark derived from standardized patient scenarios to evaluate LLMs as dynamic clinical agents across multi-turn encounters. Results show even the best model (GPT-5.5) completes only 60.4% of expert rubric items, suggesting current LLMs are not yet reliable enough for clinical practice.
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
This paper introduces AI-MASLD, a stress-audit framework for medical LLMs that reveals how benchmark accuracy can hide serious safety failures, and demonstrates that open-weight models can match or exceed proprietary ones on safety dimensions.
Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?
This paper presents a deployment-oriented stress-testing framework to evaluate how well large language models identify side effects of breast cancer radiation treatments. The study highlights limitations in LLM reliability, such as sensitivity to minor documentation changes and under-recall of rare side effects, suggesting that grounding outputs in clinician-curated lists improves robustness.
MEDSYN: Benchmarking Multi-Evidence Synthesis in Complex Clinical Cases for Multimodal Large Language Models
MEDSYN is a multilingual multimodal benchmark for evaluating MLLMs on complex clinical cases with up to 7 distinct visual evidence types per case. The study reveals that while frontier models match human experts on differential diagnosis generation, all MLLMs show significant gaps in final diagnosis selection due to poor synthesis of heterogeneous clinical evidence.
A Four-Condition Diagnostic Protocol for Evidence Utilization in Long-Context and Retrieval-Augmented Language Models
This paper introduces a four-condition diagnostic protocol to separate no-evidence answerability, oracle-evidence recoverability, full-context utilization, and retrieval-conditioned utilization in long-context and retrieval-augmented language models, tested on five open-weight models across multiple datasets.