LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis

Hugging Face Daily Papers Papers
multi-agent benchmark llm psychiatric-diagnosis chinese mental-health kdd

Summary

Introduces LingxiDiagBench, a large-scale multi-agent benchmark for evaluating LLMs on Chinese psychiatric consultation and diagnosis. Key findings show high accuracy on binary classification but poor performance on multi-way differential diagnosis, highlighting a decoupling between conversational quality and diagnostic accuracy.

Mental disorders are highly prevalent worldwide, but the shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment. Progress in AI-assisted psychiatric diagnosis is constrained by the absence of benchmarks that simultaneously provide realistic patient simulation, clinician-verified diagnostic labels, and support for dynamic multi-turn consultation. We present LingxiDiagBench, a large-scale multi-agent benchmark that evaluates LLMs on both static diagnostic inference and dynamic multi-turn psychiatric consultation in Chinese. At its core is LingxiDiag-16K, a dataset of 16,000 EMR-aligned synthetic consultation dialogues designed to reproduce real clinical demographic and diagnostic distributions across 12 ICD-10 psychiatric categories. Through extensive experiments across state-of-the-art LLMs, we establish key findings: (1) although LLMs achieve high accuracy on binary depression--anxiety classification (up to 92.3%), performance deteriorates substantially for depression--anxiety comorbidity recognition (43.0%) and 12-way differential diagnosis (28.5%); (2) dynamic consultation often underperforms static evaluation, indicating that ineffective information-gathering strategies significantly impair downstream diagnostic reasoning; (3) consultation quality assessed by LLM-as-a-Judge shows only moderate correlation with diagnostic accuracy, suggesting that well-structured questioning alone does not ensure correct diagnostic decisions. We release LingxiDiag-16K and the full evaluation framework to support reproducible research at https://github.com/Lingxi-mental-health/LingxiDiagBench.
Original Article
View Cached Full Text

Cached at: 06/24/26, 09:47 AM

Paper page - LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis

Source: https://huggingface.co/papers/2602.09379

https://huggingface.co/papers/2602.09379#lingxidiagbench-benchmarking-llms-for-chinese-psychiatric-consultation-and-diagnosis-accepted-by-kdd-2026LingxiDiagBench: Benchmarking LLMs for Chinese Psychiatric Consultation and Diagnosis [Accepted by KDD 2026]

**TL;DR:**A large-scale multi-agent benchmark revealing that while LLMs can distinguish depression from anxiety with 92.3% accuracy, they struggle badly at 12-way differential diagnosis (28.5%) — and better conversational quality doesn’t guarantee better diagnosis.

image

https://huggingface.co/papers/2602.09379#dataset-link-httpshuggingfacecodatasetsxushihao6715lingxidiag-16kDataset Link:https://huggingface.co/datasets/XuShihao6715/LingxiDiag-16K

https://huggingface.co/papers/2602.09379#the-problemThe Problem

Mental health care faces a global workforce crisis. Psychiatric diagnosis depends on nuanced, multi-turn clinical interviews, yet existing AI benchmarks fall short in three key ways: they use template-based synthetic dialogues with little variability, omit the information needed for differential diagnosis, and rarely support dynamic multi-turn consultation evaluation.

https://huggingface.co/papers/2602.09379#whats-newWhat’s New

This paper introducesLingxiDiagBench, the first large-scale, real-data-driven, multi-disease diagnostic benchmark for Chinese psychiatric AI. At its core isLingxiDiag-16K— 16,000 synthetic consultation dialogues generated from 1,709 real outpatient EMRs collected at Shanghai Mental Health Center, carefully preserving real clinical demographic and diagnostic distributions across 12 ICD-10 categories.

The benchmark coverstwo evaluation paradigms:

  • **Static:**Fixed dialogue transcripts for reproducible diagnosis and next-question prediction tasks
  • **Dynamic:**Real-time multi-turn consultation where LLMs act as Doctor Agents interviewing LLM-powered Patient Agents

Four doctor consultation strategies are compared:Free-form,Symptom-Tree,APA-Guided, andAPA-Guided + MRD-RAG.

https://huggingface.co/papers/2602.09379#key-findingsKey Findings

  • 🟢Binary classification(depression vs. anxiety) is largely solved — top models hit92.3% accuracy
  • 🟡4-way classification(including comorbidity) drops to43.0%— comorbidity recognition remains hard
  • 🔴12-way differential diagnosishits only28.5%— a substantial open challenge
  • ⚠️**Dynamic < Static:**Interactive consultation consistently underperforms static evaluation, suggesting poor information-gathering strategies hurt downstream reasoning
  • 🔍Consultation quality ≠ Diagnostic accuracy:LLM-as-a-Judge scores correlate with diagnostic accuracy at onlyr = 0.43, showing that asking good questions and reaching correct diagnoses are decoupled skills
  • ✅**RAG helps:**APA-Guided + MRD-RAG improves overall classification by ~5% over APA-Guided alone

https://huggingface.co/papers/2602.09379#why-it-mattersWhy It Matters

LingxiDiagBench provides a standardized, reproducible platform to systematically evaluate and improve AI psychiatric diagnosis — something the field has been missing. The benchmark design is language-agnostic and grounded in international clinical standards (DSM-5/ICD-10), making it extensible beyond Chinese.

https://huggingface.co/papers/2602.09379#benchmark-results-takewaysBenchmark Results Takeways

https://huggingface.co/papers/2602.09379#%F0%9F%93%8A-static-evaluation–best-model-per-task📊 Static Evaluation — Best Model per Task

Performance on fixed consultation transcripts across both the synthetic (LingxiDiag-16K) and real clinical (LingxiDiag-Clinical) test sets:

TaskBest Model (Synthetic)Acc (Synthetic)Best Model (Real)Acc (Real)2-class (Depression vs. Anxiety)Gemini-3-Flash0.854Qwen3-4B0.8874-class (+ Comorbidity + Others)Grok-4.1-Fast0.470Qwen3-32B0.52412-class (Full ICD-10 Differential)GPT-5-Mini0.409TF-IDF + SVM0.32012-class Top-3 AccuracyTF-IDF + LR0.645Qwen3-4B0.698Overall ScoreTF-IDF + LR0.533Qwen3-32B0.548

https://huggingface.co/papers/2602.09379#%F0%9F%A4%96-dynamic-evaluation–best-strategy-per-dataset🤖 Dynamic Evaluation — Best Strategy per Dataset

Performance of the end-to-end consultation pipeline (Doctor Agent → Patient Agent → Diagnosis), across both data settings:

StrategyBest Model2-class Acc4-class Acc12-class AccClf-Ovl**Synthetic (LingxiDiag-16K)Free-formGrok-4.1-Fast88.6%34.0%25.5%40.1%Symptom-TreeDeepSeek-V3.286.5%31.0%21.5%38.0%APA-GuidedDeepSeek-V3.288.5%31.5%23.0%41.2%APA-Guided + MRD-RAGGrok-4.1-Fast****88.5%****43.0%****28.5%****45.4%Real (LingxiDiag-Clinical)Free-formQwen3-8B88.8%40.0%43.0%49.0%Symptom-TreeGPT-OSS-20B91.2%****43.0%****44.5%****50.0%**APA-GuidedQwen3-32B80.0%36.0%46.5%48.3%APA-Guided + MRD-RAGGPT-OSS-20B78.8%37.5%45.5%47.2%

https://huggingface.co/papers/2602.09379#%F0%9F%94%81-cross-dataset-transfer–does-synthetic-training-generalize-to-real-data🔁 Cross-Dataset Transfer — Does Synthetic Training Generalize to Real Data?

To validate that LingxiDiag-16K encodes clinically meaningful knowledge (not just surface statistics), models fine-tuned on synthetic data were evaluated on real clinical cases:

Model12-class Acc (Real, Zero-shot)12-class Acc (Real, +LoRA SFT)GainQwen3-8B4.1%41.4%+37.3%Qwen3-32B20.4%39.7%+19.3% The authors emphasize that this benchmark is for research purposes only and must not be deployed in clinical settings without rigorous validation and human oversight.

Similar Articles

IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

arXiv cs.AI

IMCBench is a new benchmark for evaluating multimodal LLMs on image-grounded medical conversations, pairing clinical images with synthetic patient profiles. Evaluations across safety, accuracy, and uncertainty show that even strong models like Claude Opus 4.6 have safety issues, highlighting the need for multi-dimensional evaluation.

DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation

arXiv cs.CL

DLawBench is a new benchmark for evaluating large language models in multi-turn legal consultation, covering Chinese and US law with four client types. Experiments show significant room for improvement, with the best model achieving only 0.562 on legal reasoning.