When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening
Summary
This paper introduces a SCID-anchored benchmark of 555 interviews to evaluate five LLMs for psychiatric screening, finding that while models show potential, they tend to discount symptom evidence in the presence of preserved functioning or protective context, requiring careful validation.
View Cached Full Text
Cached at: 05/25/26, 08:59 AM
# When Symptoms Are Not Enough: Evidence-Weighting Patterns in Large Language Model Psychiatric Screening Source: [https://arxiv.org/abs/2605.23148](https://arxiv.org/abs/2605.23148) [View PDF](https://arxiv.org/pdf/2605.23148) > Abstract:As demand for mental health care outpaces clinician\-delivered assessment, scalable screening tools are increasingly needed\. Large language models \(LLMs\) may identify psychiatric risk from patient narratives, but their reliability across diagnoses, demographic subgroups, and evidence\-use patterns remains uncertain\. We introduce a SCID\-anchored benchmark of 555 semi\-structured experiential interviews paired with diagnostic reference labels for anxiety disorder, major depressive disorder, post\-traumatic stress disorder, and any current mental health disorder\. Using zero\-shot task\-specific prompting, we evaluated five state\-of\-the\-art LLMs and examined whether false\-negative errors reflected missed psychiatric evidence or differential weighting of symptom, functional\-impairment, and protective\-context cues\. Performance varied across tasks and models, with accuracy ranging from 0\.49 to 0\.86 and Matthews correlation coefficients from 0\.16 to 0\.38\. GPT\-4\.1 Mini and GPT\-5 Mini showed the most consistent disorder\-specific accuracy\. Subgroup analyses found higher depression\-classification accuracy among male than female participants, no consistent age\-related pattern, and modest non\-uniform variation across race strata\. Evidence\-integration analyses showed that false\-negative anxiety and PTSD classifications often contained explicit symptom evidence but were accompanied by preserved functioning, coping ability, or social support\. Functional\-impairment evidence shifted model outputs toward positive classifications, whereas protective\-context evidence shifted outputs away\. These findings suggest that LLMs may support scalable psychiatric screening, but their tendency to discount symptom evidence in the presence of preserved functioning or protective context requires careful validation before clinical deployment\. ## Submission history From: Jianfeng Zhu \[[view email](https://arxiv.org/show-email/c8efb6c7/2605.23148)\] **\[v1\]**Fri, 22 May 2026 01:53:03 UTC \(1,302 KB\)
Similar Articles
Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry
Introduces Safe-Psych, a sequential benchmark for evaluating how large language models handle diagnostic uncertainty in psychiatry, revealing that even strong models often fail to abstain or seek clarification when clinical evidence is incomplete.
LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis
Introduces LingxiDiagBench, a large-scale multi-agent benchmark for evaluating LLMs on Chinese psychiatric consultation and diagnosis. Key findings show high accuracy on binary classification but poor performance on multi-way differential diagnosis, highlighting a decoupling between conversational quality and diagnostic accuracy.
Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases
Researchers introduce MedSP1000, a 1,638-case interactive benchmark derived from standardized patient scenarios to evaluate LLMs as dynamic clinical agents across multi-turn encounters. Results show even the best model (GPT-5.5) completes only 60.4% of expert rubric items, suggesting current LLMs are not yet reliable enough for clinical practice.
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy
KlinikeBench introduces a clinician-verified benchmark of 333 interactive clinical tasks where LMs must gather patient history, use clinical tools, and reach a diagnosis within a turn budget. Across 31 models, the best (e.g., GPT-6-astra, Claude Opus 5) succeed on under 30% of tasks despite 90.7% diagnostic accuracy, revealing a large gap between diagnostic knowledge and real-world clinical interaction.
Do Language Models Know What Compelling Scientific Evidence Looks Like?
The paper introduces EDB-Bio, a 96k-episode benchmark for scientific evidence deliberation across three biological domains, showing that frontier and open-weight models are systematically underconfidence-prone (Brier scores near a trivial base-rate forecaster) and that post-training with proper scoring rules improves calibration, yielding an Ibis0 model 2.5× more efficient than the best frontier model in sequential experiment selection.