MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

# MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
Source: [https://arxiv.org/html/2607.08257](https://arxiv.org/html/2607.08257)
ModelObjective Comparison \(%\)Subjective Assessment \(1–5\)Inter\.Exam\.NoteCate\.Diso\.Emp\.Prof\.Qual\.RigorApp\.\\rowcolorRowGray Medical Trainees78\.2371\.0582\.9181\.4272\.581\.633\.863\.872\.332\.47Human Experts82\.3591\.9088\.6790\.5983\.241\.224\.254\.012\.542\.96\\rowcolorGroupGraySmall Language ModelsLlama\-3\-8B19\.067\.949\.9618\.116\.082\.052\.881\.931\.862\.67\\rowcolorRowGray Qwen3\-14B23\.198\.3312\.1223\.5515\.191\.962\.902\.172\.202\.83GPT\-OSS\-20B27\.438\.6033\.1237\.2125\.811\.603\.153\.142\.063\.23\\rowcolorRowGray QWQ\-32B30\.4010\.8732\.3044\.5326\.392\.133\.722\.853\.143\.40\\rowcolorGroupGrayLarge Language ModelsGPT\-OSS\-120B36\.729\.9239\.9449\.5434\.191\.503\.453\.892\.423\.35\\rowcolorRowGray Claude\-Sonnet\-4\.630\.4421\.2834\.8150\.2834\.571\.723\.392\.702\.223\.17Deepseek\-v4\-Pro30\.9521\.9356\.4626\.2020\.852\.093\.482\.912\.393\.29\\rowcolorRowGray GPT\-5\.569\.0622\.9945\.4470\.2349\.851\.503\.633\.662\.213\.48\\rowcolorGroupGrayMedical\-specific ModelsMedGemma\-4B\-it25\.659\.4210\.6721\.483\.052\.162\.571\.992\.152\.91\\rowcolorRowGray MedGemma\-27B\-it43\.1216\.0535\.9132\.2710\.102\.052\.832\.672\.333\.17Baichuan\-M251\.2924\.7650\.5480\.5643\.191\.292\.852\.912\.443\.33

## 4Experiments

##### Task Definition\.

We formulate evaluation as a multi\-stage, mixed\-type clinical task\. Given a casecc, a doctor agent completes an episode covering subjective interviewing, objective examination, diagnostic assessment, and treatment planning\. The task combines three forms: \(1\) open\-ended generation, including dialogue, clinical\-note writing, diagnostic reasoning, and treatment planning; \(2\) multi\-label decision making, including examination recommendation and category\-level diagnosis; and \(3\) ranked disorder prediction, for definitive diagnosis\. We evaluate the resulting trajectory and stage outputs from two perspectives: \(i\) objective comparison, which measures EHR\-grounded outcomes, and \(ii\) subjective assessment, which measures process\-level clinical quality\.

##### Evaluation Metrics\.

For EHR\-grounded outputs, including examination recommendation \(Exam\.\), clinical interviewing \(Inter\.\), and clinical\-note generation \(Note\), we measure reference coverage:

Coverage​\(yc,𝒦c\)=1\|𝒦c\|​∑k∈𝒦c𝕀​\[k⪯yc\],\\mathrm\{Coverage\}\(y\_\{c\},\\mathcal\{K\}\_\{c\}\)=\\frac\{1\}\{\|\\mathcal\{K\}\_\{c\}\|\}\\sum\_\{k\\in\\mathcal\{K\}\_\{c\}\}\\mathbb\{I\}\\left\[k\\preceq y\_\{c\}\\right\],\(13\)where𝒦c\\mathcal\{K\}\_\{c\}denotes the EHR\-derived reference checkpoint set for casecc, andk⪯yck\\preceq y\_\{c\}indicates that checkpointkkis semantically entailed by the generated outputycy\_\{c\}\. We use a conservative semantic similarity threshold ofτ=0\.85\\tau=0\.85for high\-precision automatic matching, and adjudicate pairs below the threshold with an empirically validated Multi\-LLM\-assisted semantic matching procedure\. For definitive diagnosis, we report category identification \(Cate\.\) and disorder\-level diagnosis ranking \(Diso\.\), with nDCG used to evaluate diagnostic prioritization\. For subjective assessment, MentalEval scores communication empathy \(Emp\.\), interviewing professionalism \(Prof\.\), clinical\-note quality \(Qual\.\), diagnostic rigor \(Rigor\), and treatment appropriateness \(App\.\) using expert\-defined rubrics\. For more details see Appendix[K](https://arxiv.org/html/2607.08257#A11)\.

##### Comparison Groups\.

We consider five comparison groups: human experts, medical trainees, crowdworkers, general\-purpose LLMs, and medical\-specific LLMs\. The expert group consists of 5 licensed psychiatrists and 3 psychology experts\. We also include 14 medical trainees and 8 non\-specialist crowdworkers\. For LLMs, we evaluate a set of general\-purpose models, including Llama\-3\-8B\[[16](https://arxiv.org/html/2607.08257#bib.bib57)\], Qwen3\-14B\[[52](https://arxiv.org/html/2607.08257#bib.bib58)\], GPT\-OSS\-20B\[[1](https://arxiv.org/html/2607.08257#bib.bib59)\], QwQ\-32B\[[52](https://arxiv.org/html/2607.08257#bib.bib58)\], GPT\-OSS\-120B\[[1](https://arxiv.org/html/2607.08257#bib.bib59)\], DeepSeek\-V4\-Pro\[[27](https://arxiv.org/html/2607.08257#bib.bib44)\], GPT\-5\.5\[[38](https://arxiv.org/html/2607.08257#bib.bib53)\], and Claude\-Sonnet\-4\-6\[[5](https://arxiv.org/html/2607.08257#bib.bib54)\]\. We also include medical\-oriented LLMs, including MedGemma\-4B\-it\[[37](https://arxiv.org/html/2607.08257#bib.bib55)\], MedGemma\-27B\-it\[[37](https://arxiv.org/html/2607.08257#bib.bib55)\], and Baichuan\-M2\[[12](https://arxiv.org/html/2607.08257#bib.bib60)\]\.

### 4\.1Can LLMs Interact Like Clinicians?

We evaluate 12 LLMs, 8 human experts, and 14 medical trainees in MentalHospital\. As shown in Table[3\.2](https://arxiv.org/html/2607.08257#S3.SS2), current LLMs still show a clear gap from clinicians in objective clinical competence\. Compared with the strongest medical\-specific model, Baichuan\-M2, medical trainees achieve higher scores across all five objective metrics, with an average gap of 27\.17 percentage points; human experts further widen this gap to 37\.28 points\. In subjective assessment, LLMs and clinicians show complementary strengths rather than uniform superiority of either group\. Human experts exhibit stronger interviewing professionalism \(4\.25\), reflecting more efficient, goal\-directed clinical information gathering, but receive lower scores in expressed communication empathy \(1\.22\)\. In contrast, the best\-performing LLMs reach or exceed human experts in expressed empathy \(2\.16 vs\. 1\.22\), diagnostic rigor \(3\.14 vs\. 2\.54\), and treatment appropriateness \(3\.48 vs\. 2\.96\), while remaining lower in interviewing professionalism \(3\.72 vs\. 4\.25\) and clinical\-note quality \(3\.89 vs\. 4\.01\)\. These results suggest that LLMs are not yet reliable substitutes for clinicians, but may provide complementary value in verbal affective support and evidence\-informed treatment assistance\.

Table 3:Coverage by clinical component\.ModelInterview PhaseNote PhaseAvg\.CCMSAvg\.CCMSQwen3\-14B23\.1944\.3612\.5712\.1216\.3817\.25\\rowcolorrowgray MedGemma\-4B\-it25\.6540\.6910\.4510\.679\.686\.66QWQ\-32B30\.4050\.9017\.8432\.3042\.9134\.22\\rowcolorrowgray Deepseek\-v4\-Pro30\.9541\.6723\.1356\.4659\.9443\.91GPT\-5\.569\.0685\.3855\.7245\.4455\.3465\.89

### 4\.2Where Do LLMs Fall Short in Clinical Interaction?

Table 4:Ablation of skill\-augmented patient construction for standardized patients\.Patient VariantObjective Comparison \(%\)Subjective Assessment \(1–5\)Cov\.↑\\uparrowPrec\.↑\\uparrowHalluc\.↓\\downarrowRealism↑\\uparrowFidelity↑\\uparrowDistinct\.↑\\uparrowNaive Prompting72\.4086\.107\.802\.122\.231\.92\\rowcolorRowGray Representation Skill Only72\.40\+0\.00\+0\.0085\.72−0\.38\-0\.389\.04\+1\.24\+1\.244\.12\+2\.00\+2\.003\.82\+1\.59\+1\.593\.72\+1\.80\+1\.80Memory Skill Only84\.83\+12\.43\+12\.4391\.64\+5\.92\+5\.924\.52−4\.52\-4\.523\.55−0\.57\-0\.573\.87\+0\.05\+0\.052\.77−0\.95\-0\.95\\rowcolorRowGray Full Patient88\.91\+4\.08\+4\.0893\.27\+1\.63\+1\.633\.08−1\.44\-1\.444\.17\+0\.62\+0\.624\.05\+0\.18\+0\.183\.87\+1\.10\+1\.10To identify the core gap, we decompose coverage in clinical interviewing and note generation into Chief Complaint \(CC\) and Mental Status \(MS\) in Table[3](https://arxiv.org/html/2607.08257#S4.T3), current LLMs fail to recognize the clinical importance of mental status examination in psychiatric assessment and diagnosis\. For example,GPT\-5\.5shows a large gap in interviewing \(85\.38 vs\. 55\.72\), andDeepSeek\-v4\-Proexhibits a similar pattern in note generation \(59\.94 vs\. 43\.91\)\.

### 4\.3Does Skill\-Augmented Patient Construction Improve Fidelity?

We evaluate patient construction fidelity through objective evaluation and subjective assessment, with detailed experimental settings provided in Appendix[L](https://arxiv.org/html/2607.08257#A12)\. As shown in Table[4](https://arxiv.org/html/2607.08257#S4.T4), naive prompting achieves reasonable factual faithfulness but performs poorly in subjective fidelity, indicating that directly exposing the EHR does not yield realistic patient simulation\. Representation skill substantially improves clinical realism and patient distinctiveness, while memory skill improves evidence coverage, precision, and hallucination control\. The full construction achieves the best overall performance\.

Table 5:Ablation results of MentalEval training against expert ratings\. DPO is applied after SFT\.EvaluatorBase Model \(Qwen3\-8B\)\+SFT\+DPOQWK↑\\uparrowAcc\.↑\\uparrowMAE↓\\downarrowQWK↑\\uparrowAcc\.↑\\uparrowMAE↓\\downarrowQWK↑\\uparrowAcc\.↑\\uparrowMAE↓\\downarrowMentalEval\-Emp0\.6770\.2250\.8750\.824\+0\.148\+0\.1480\.500\+0\.275\+0\.2750\.525−0\.350\-0\.3500\.926\+0\.102\+0\.1020\.725\+0\.225\+0\.2250\.275−0\.250\-0\.250\\rowcolorRowGray MentalEval\-Pro0\.7050\.2750\.8250\.844\+0\.139\+0\.1390\.550\+0\.275\+0\.2750\.475−0\.350\-0\.3500\.941\+0\.097\+0\.0970\.775\+0\.225\+0\.2250\.225−0\.250\-0\.250MentalEval\-Note0\.7660\.4000\.6750\.910\+0\.144\+0\.1440\.700\+0\.300\+0\.3000\.300−0\.375\-0\.3750\.940\+0\.030\+0\.0300\.775\+0\.075\+0\.0750\.225−0\.075\-0\.075\\rowcolorRowGray MentalEval\-Diag0\.8280\.4750\.5750\.887\+0\.059\+0\.0590\.725\+0\.250\+0\.2500\.325−0\.250\-0\.2500\.967\+0\.080\+0\.0800\.875\+0\.150\+0\.1500\.125−0\.200\-0\.200MentalEval\-Treat0\.8190\.4250\.6000\.928\+0\.109\+0\.1090\.725\+0\.300\+0\.3000\.275−0\.325\-0\.3250\.948\+0\.020\+0\.0200\.800\+0\.075\+0\.0750\.200−0\.075\-0\.075Average0\.7590\.3600\.7100\.879\+0\.120\+0\.1200\.640\+0\.280\+0\.2800\.380−0\.330\-0\.3300\.944\+0\.066\+0\.0660\.790\+0\.150\+0\.1500\.210−0\.170\-0\.170

![Refer to caption](https://arxiv.org/html/2607.08257v1/x3.png)Figure 3:Agreement with expert ratings, where MentalEval shows the strongest alignment\.
### 4\.4Does MentalEval Align with Expert Judgment?

MentalEval is designed as a task\-specific evaluator for MentalHospital, rather than a general\-purpose LLM judge across arbitrary clinical tasks\. Our goal is therefore to test whether it can reproduce expert judgments within the MentalHospital evaluation setting\.To avoid evaluation leakage, we additionally sampled 40 psychiatric EHR cases from the coordinating medical center to construct the MentalEval test set\. These cases were strictly excluded from all MentalEval training data\. For each case, we sample model\-generated responses and evaluate them across the five MentalEval dimensions, yielding 200 dimension\-level test samples in total\. Each sample is independently rated by three experts, and only unanimously assigned samples are retained as expert\-validated references\. We ablate the two\-stage training procedure on this held\-out set of unseen expert\-labeled cases\. As shown in Table[5](https://arxiv.org/html/2607.08257#S4.T5), rubric\-grounded SFT consistently improves all five evaluators, increasing average QWK from 0\.759 to 0\.879 and reducing MAE from 0\.710 to 0\.380\. Expert\-guided DPO further improves agreement, reaching an average QWK of 0\.944, accuracy of 0\.790, and MAE of 0\.210\. We then compare the trained MentalEval with LLM\-as\-a\-Judge, Multi\-LLM Jury, and individual crowdworker ratings in Figure[3](https://arxiv.org/html/2607.08257#S4.F3)\. MentalEval achieves the strongest expert alignment, with the highest QWK \(0\.926\) and accuracy \(0\.725\), and the lowest MAE \(0\.275\)\.

### 4\.5Does MentalHospital Provide a Clinically Faithful Environment?

![Refer to caption](https://arxiv.org/html/2607.08257v1/x4.png)Figure 4:Clinical fidelity ratings for MentalHospital\.We conduct a survey to assess the clinical faithfulness with 5 psychiatrists, 3 psychologists, and 14 medical trainees \(full questionnaire is see Appendix[M](https://arxiv.org/html/2607.08257#A13)\)\. The questionnaire covers five dimensions: clinical realism \(CR\), patient fidelity \(PF\), patient distinctiveness \(PD\), evaluation credibility \(EC\), and training suitability \(TS\), using a 5\-point Likert scale\. As shown in Figure[4](https://arxiv.org/html/2607.08257#S4.F4), MentalHospital receives consistently positive ratings, with an overall mean score of 3\.88/5 \(95% bootstrap CI: \[3\.78, 3\.98\]\), a positive response rate of 69\.9% \(95% CI: \[64\.8%, 75\.0%\]\), and a negative response rate of only 5\.1% \(95% CI: \[3\.4%, 6\.8%\]\)\. The ratings are particularly strong for training suitability and evaluation credibility, with positive response rates of 84\.8% and 75\.0%, suggesting that MentalHospital provides a clinically plausible environment\.

### 4\.6Qualitative Error Analysis

![Refer to caption](https://arxiv.org/html/2607.08257v1/x5.png)Figure 5:Representative cases in MentalHospital\.Figure[5](https://arxiv.org/html/2607.08257#S4.F5)illustrates two representative failure modes observed in MentalHospital\. In a case involving anxiety and possible perceptual disturbance,GPT\-5\.5directly probes hallucination\-related features such as source and content, yielding higher objective evidence coverage thanQwen3\-14B\. However, its response contains limited empathic acknowledgment, resulting in a lower MentalEval empathy score\. Conversely,Qwen3\-14Bprovides more explicit emotional validation and reassurance, but fails to pursue the hallucination hypothesis through targeted psychiatric questioning\. This contrast shows that evidence\-seeking behavior and process\-level communication quality can diverge substantially\. A model may recover key diagnostic evidence while offering poor affective support, whereas another may appear empathic but remain clinically under\-informative\.

## 5Related Work

##### Psychiatric Benchmarks\.

Early medical LLM evaluation mainly relied on static question\-answering benchmarks, testing medical knowledge recall and exam\-style clinical reasoning\[[40](https://arxiv.org/html/2607.08257#bib.bib16),[46](https://arxiv.org/html/2607.08257#bib.bib17)\]\. This paradigm was later expanded to broader medical evaluation suites and real\-world clinical benchmarks covering diverse tasks, safety, and clinical reliability\[[9](https://arxiv.org/html/2607.08257#bib.bib19),[7](https://arxiv.org/html/2607.08257#bib.bib18),[56](https://arxiv.org/html/2607.08257#bib.bib20),[4](https://arxiv.org/html/2607.08257#bib.bib21)\], with subsequent work further examining clinical safety, expert consensus, and specialized reasoning\[[47](https://arxiv.org/html/2607.08257#bib.bib22),[31](https://arxiv.org/html/2607.08257#bib.bib23),[51](https://arxiv.org/html/2607.08257#bib.bib24),[58](https://arxiv.org/html/2607.08257#bib.bib27)\]\. In parallel, mental\-health benchmarks began to assess counseling, emotional support, and psychotherapy\[[25](https://arxiv.org/html/2607.08257#bib.bib28),[49](https://arxiv.org/html/2607.08257#bib.bib26),[55](https://arxiv.org/html/2607.08257#bib.bib6)\], while clinically oriented psychiatric benchmarks focused on psychiatric practice, disorder recognition, and diagnostic reasoning\[[30](https://arxiv.org/html/2607.08257#bib.bib29),[14](https://arxiv.org/html/2607.08257#bib.bib30),[42](https://arxiv.org/html/2607.08257#bib.bib5),[41](https://arxiv.org/html/2607.08257#bib.bib31)\]\. These efforts connect subclinical mental\-health support with threshold\-level psychiatric assessment, but still largely evaluate isolated responses or task\-specific outputs\. Recent work has moved toward multi\-turn dialogue and simulated clinical environments, testing contextual interaction across turns\[[54](https://arxiv.org/html/2607.08257#bib.bib32),[35](https://arxiv.org/html/2607.08257#bib.bib33),[29](https://arxiv.org/html/2607.08257#bib.bib34),[24](https://arxiv.org/html/2607.08257#bib.bib35),[53](https://arxiv.org/html/2607.08257#bib.bib36),[34](https://arxiv.org/html/2607.08257#bib.bib62)\]and patient–doctor simulation\[[36](https://arxiv.org/html/2607.08257#bib.bib11),[13](https://arxiv.org/html/2607.08257#bib.bib10),[28](https://arxiv.org/html/2607.08257#bib.bib12),[43](https://arxiv.org/html/2607.08257#bib.bib38),[3](https://arxiv.org/html/2607.08257#bib.bib37),[50](https://arxiv.org/html/2607.08257#bib.bib51)\]\. However, existing benchmarks still rarely support virtual psychiatric encounters\.

##### Evaluation Methods\.

Early evaluation of LLM outputs mainly relied on task accuracy and surface\-level automatic metrics, including lexical overlap and embedding\-based similarity\[[32](https://arxiv.org/html/2607.08257#bib.bib39),[26](https://arxiv.org/html/2607.08257#bib.bib40),[6](https://arxiv.org/html/2607.08257#bib.bib41),[57](https://arxiv.org/html/2607.08257#bib.bib42)\]\. These metrics remain common in benchmark\-based assessment and text generation evaluation\[[39](https://arxiv.org/html/2607.08257#bib.bib7),[56](https://arxiv.org/html/2607.08257#bib.bib20)\], but they are insufficient for clinically nuanced behaviors such as empathy, professionalism, diagnostic rigor, and treatment appropriateness\. Later work therefore incorporated clinician\-centered or expert\-based protocols\[[19](https://arxiv.org/html/2607.08257#bib.bib43),[4](https://arxiv.org/html/2607.08257#bib.bib21)\], with similar trends in clinical safety and reasoning benchmarks\[[47](https://arxiv.org/html/2607.08257#bib.bib22),[59](https://arxiv.org/html/2607.08257#bib.bib45)\]\. Although expert review improves clinical validity, it is difficult to scale\. Recent studies have explored model\-based judging, including judge\-style, jury\-style, and rubric\-driven frameworks\[[10](https://arxiv.org/html/2607.08257#bib.bib15),[7](https://arxiv.org/html/2607.08257#bib.bib18),[23](https://arxiv.org/html/2607.08257#bib.bib46)\], while also highlighting reliability concerns of LLM\-based judges in medical settings\[[8](https://arxiv.org/html/2607.08257#bib.bib47),[48](https://arxiv.org/html/2607.08257#bib.bib48),[17](https://arxiv.org/html/2607.08257#bib.bib49)\]\. Existing methods therefore still struggle to balance expertise and scalability, especially for subjective mental\-health evaluation where process quality cannot be reduced to objective correctness alone\.

## 6Conclusion

We present MentalHospital, a virtual psychiatric evaluation environment for LLM\-based medical AI\. MentalHospital supports interactive S\.O\.A\.P\.\-based psychiatric assessment with EHR\-grounded virtual patients and a dual\-track evaluation protocol that combines objective comparison with scalable subjective assessment\. To operationalize specialist judgment, we develop MentalEval, a suite of five domain\-specific evaluators trained through rubric\-grounded SFT and expert\-guided DPO\. Experiments show that current LLMs remain limited in clinician\-like psychiatric interaction, especially in mental status assessment, while MentalHospital and MentalEval provide faithful and scalable tools for evaluating in psychiatric care\. Finally, clinician survey responses indicate positive acceptance of MentalHospital and its perceived value for psychiatric training\.

## References

- \[1\]S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[2\]\(2025\)The evaluation illusion of large language models in medicine\.npj Digital Medicine8\(1\),pp\. 600\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p2.1)\.
- \[3\]M\. Almansoori, K\. Kumar, and H\. Cholakkal\(2025\)Self\-evolving multi\-agent simulations for realistic clinical interactions\.arXiv preprint arXiv:2503\.22678\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[4\]R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel,et al\.\(2025\)Healthbench: evaluating large language models towards improved human health\.arXiv preprint arXiv:2505\.08775\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.3.1),[§1](https://arxiv.org/html/2607.08257#S1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[5\]Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§3\.1](https://arxiv.org/html/2607.08257#S3.SS1.p1.2),[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[6\]S\. Banerjee and A\. Lavie\(2005\)METEOR: an automatic metric for mt evaluation with improved correlation with human judgments\.InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization,pp\. 65–72\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[7\]S\. Bedi, H\. Cui, M\. Fuentes, A\. Unell, M\. Wornow, J\. M\. Banda, N\. Kotecha, T\. Keyes, Y\. Mai, M\. Oez,et al\.\(2025\)Medhelm: holistic evaluation of large language models for medical tasks\.arXiv preprint arXiv:2505\.23802\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[8\]I\. Belmadani, O\. El Khettari, P\. C\. dit Beaufils, R\. Dufour, and B\. Favre\(2026\)Who judges the judge? evaluating llm\-as\-a\-judge for french medical open\-ended qa\.InProceedings of the 1st Workshop on Linguistic Analysis for Health \(HeaLing 2026\),pp\. 142–157\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[9\]P\. Chen, J\. Ye, G\. Wang, Y\. Li, Z\. Deng, W\. Li, T\. Li, H\. Duan, Z\. Huang, Y\. Su,et al\.\(2024\)Gmai\-mmbench: a comprehensive multimodal evaluation benchmark towards general medical ai\.Advances in Neural Information Processing Systems37,pp\. 94327–94427\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[10\]E\. Croxford, Y\. Gao, E\. First, N\. Pellegrino, M\. Schnier, J\. Caskey, M\. Oguss, G\. Wills, G\. Chen, D\. Dligach,et al\.\(2025\)Evaluating clinical ai summaries with large language models as judges\.npj Digital Medicine8\(1\),pp\. 640\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[11\]J\. Donohoe\(2015\)Implementing an education programme and soap notes framework to improve nursing documentation\.Ph\.D\. Thesis,Royal College of Surgeons in Ireland\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p3.1)\.
- \[12\]C\. Dou, C\. Liu, F\. Yang, F\. Li, J\. Jia, M\. Chen, Q\. Ju, S\. Wang, S\. Dang, T\. Li,et al\.\(2025\)Baichuan\-m2: scaling medical capability with large verifier system\.arXiv preprint arXiv:2509\.02208\.Cited by:[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[13\]Z\. Fan, L\. Wei, J\. Tang, W\. Chen, W\. Siyuan, Z\. Wei, and F\. Huang\(2025\)Ai hospital: benchmarking large language models in a multi\-agent medical interaction simulator\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 10183–10213\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.8.1),[§1](https://arxiv.org/html/2607.08257#S1.p2.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[14\]A\. E\. Fouda, A\. A\. Hassan, R\. J\. Hanafy, and M\. E\. Fouda\(2025\)PsychiatryBench: a multi\-task benchmark for llms in psychiatry\.arXiv preprint arXiv:2509\.09711\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[15\]L\. Gong, W\. Fang, T\. Yang, D\. Tao, C\. Guo, P\. Wei, B\. Xie, J\. Guan, Z\. Chen, F\. Shi,et al\.\(2026\)MedDialogRubrics: a comprehensive benchmark and evaluation framework for multi\-turn medical consultations in large language models\.arXiv preprint arXiv:2601\.03023\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.4.1)\.
- \[16\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[17\]J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu,et al\.\(2024\)A survey on llm\-as\-a\-judge\.The Innovation\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[18\]Y\. Jiang, K\. C\. Black, G\. Geng, D\. Park, J\. Zou, A\. Y\. Ng, and J\. H\. Chen\(2025\)MedAgentBench: a virtual ehr environment to benchmark medical llm agents\.Nejm Ai2\(9\),pp\. AIdbp2500144\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p2.1)\.
- \[19\]V\. Kocaman, M\. A\. Kaya, A\. M\. Feier, and D\. Talby\(2025\)Clinical large language model evaluation by expert review \(clever\): framework development and validation\.JMIR AI4\(1\),pp\. e72153\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[20\]S\. Kolding, R\. M\. Lundin, L\. Hansen, and S\. D\. Østergaard\(2025\)Use of generative artificial intelligence \(ai\) in psychiatry and mental health care: a systematic review\.Acta Neuropsychiatrica37,pp\. e37\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1)\.
- \[21\]E\. E\. Lee, J\. Torous, M\. De Choudhury, C\. A\. Depp, S\. A\. Graham, H\. Kim, M\. P\. Paulus, J\. H\. Krystal, and D\. V\. Jeste\(2021\)Artificial intelligence for mental health care: clinical applications, barriers, facilitators, and artificial wisdom\.Biological Psychiatry: Cognitive Neuroscience and Neuroimaging6\(9\),pp\. 856–864\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1)\.
- \[22\]J\. Lee, K\. Lim, Y\. Jung, and B\. Kim\(2025\)Psyche: a multi\-faceted patient simulation framework for evaluation of psychiatric assessment conversational agents\.arXiv preprint arXiv:2501\.01594\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.7.1)\.
- \[23\]Y\. Lee, J\. Kim, J\. Kim, H\. Cho, J\. Kang, P\. Kang, and N\. Kim\(2025\)Checkeval: a reliable llm\-as\-a\-judge framework for evaluating text generation using checklists\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 15782–15809\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[24\]S\. S\. Li, V\. Balachandran, S\. Feng, J\. S\. Ilgen, E\. Pierson, P\. W\. Koh, and Y\. Tsvetkov\(2024\)Mediq: question\-asking llms and a benchmark for reliable interactive clinical reasoning\.Advances in Neural Information Processing Systems37,pp\. 28858–28888\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[25\]Y\. Li, J\. Yao, J\. B\. S\. Bunyi, A\. C\. Frank, A\. Hwang, and R\. Liu\(2025\)Counselbench: a large\-scale expert evaluation and adversarial benchmark of large language models in mental health counseling\.arXiv e\-prints,pp\. arXiv–2506\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p2.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[26\]C\. Lin\(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[27\]A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.\(2025\)Deepseek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[Appendix F](https://arxiv.org/html/2607.08257#A6.SS0.SSS0.Px3.p1.1),[Appendix F](https://arxiv.org/html/2607.08257#A6.SS0.SSS0.Px6.p1.1),[§2\.2](https://arxiv.org/html/2607.08257#S2.SS2.SSS0.Px3.p1.4),[§2](https://arxiv.org/html/2607.08257#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.08257#S3.SS1.p1.2),[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[28\]P\. Liu, R\. Ding, Y\. Mao, Z\. Jiang, Y\. Ye, Y\. Gao, Y\. Zhang, R\. Sun, L\. Lai, and Z\. Qian\(2026\)OpenHospital: a thing\-in\-itself arena for evolving and benchmarking llm\-based collective intelligence\.arXiv preprint arXiv:2603\.14771\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.9.1),[§1](https://arxiv.org/html/2607.08257#S1.p2.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[29\]R\. Liu, K\. Xue, X\. Zhang, and S\. Zhang\(2025\)Interactive evaluation for medical llms via task\-oriented dialogue system\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 4871–4896\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[30\]S\. Liu, R\. Wang, L\. Zhang, X\. Zhu, R\. Yang, X\. Zhou, F\. Wu, Z\. Yang, C\. Jin, and G\. Wang\(2025\)PsychBench: a comprehensive and professional benchmark for evaluating the performance of llm\-assisted psychiatric clinical practice\.arXiv preprint arXiv:2503\.01903\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[31\]N\. Mehandru, N\. Golchini, D\. Bamman, T\. Zack, M\. F\. Molina, and A\. Alaa\(2025\)Er\-reason: a benchmark dataset for llm\-based clinical reasoning in the emergency room\.arXiv preprint arXiv:2505\.22919\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[32\]K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu\(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[33\]R\. Patel, S\. N\. Wee, R\. Ramaswamy, S\. Thadani, G\. Guruswamy, R\. Garg, N\. Calvanese, M\. Valko, A\. Rush, M\. Rentería,et al\.\(2022\)NeuroBlu: a natural language processing \(nlp\) electronic health record \(ehr\) data analytic tool to generate real\-world evidence in mental healthcare\.European Psychiatry65\(S1\),pp\. S99–S100\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1)\.
- \[34\]J\. Pombal, M\. D’Eon, N\. M\. Guerreiro, P\. H\. Martins, A\. Farinhas, and R\. Rei\(2025\)MindEval: benchmarking language models on multi\-turn mental health support\.arXiv preprint arXiv:2511\.18491\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[35\]S\. Racha, P\. H\. Joshi, U\. Maurya, N\. Yadav, M\. Sharma, A\. Kunisetty, S\. Darisipudi, N\. Punjabi, and G\. Ramakrishnan\(2026\)OMIND: framework for knowledge grounded finetuning and multi\-turn dialogue benchmark for mental health llms\.arXiv preprint arXiv:2603\.25105\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[36\]S\. Schmidgall, R\. Ziaei, C\. Harris, E\. Reis, J\. Jopling, and M\. Moor\(2024\)Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments\.arXiv preprint arXiv:2405\.07960\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.10.1),[§1](https://arxiv.org/html/2607.08257#S1.p2.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[37\]A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.\(2025\)Medgemma technical report\.arXiv preprint arXiv:2507\.05201\.Cited by:[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[38\]A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§3\.1](https://arxiv.org/html/2607.08257#S3.SS1.p1.2),[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[39\]K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[40\]K\. Singhal, T\. Tu, J\. Gottweis, R\. Sayres, E\. Wulczyn, M\. Amin, L\. Hou, K\. Clark, S\. R\. Pfohl, H\. Cole\-Lewis,et al\.\(2025\)Toward expert\-level medical question answering with large language models\.Nature medicine31\(3\),pp\. 943–950\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[41\]H\. Song, M\. Kang, J\. Shin, J\. Kim, C\. Park, H\. Yoo, J\. An, A\. Oh, J\. Han, and K\. Lim\(2026\)MentalBench: a benchmark for evaluating psychiatric diagnostic capability of large language models\.arXiv preprint arXiv:2602\.12871\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[42\]X\. Sun, Y\. Yang, J\. Zhu, J\. Zhong, X\. Zhou, and K\. Wei\(2026\)MentalSeek\-dx: towards progressive hypothetico\-deductive reasoning for real\-world psychiatric diagnosis\.arXiv preprint arXiv:2602\.03340\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[43\]I\. Sviridov, A\. Miftakhova, T\. A\. Vladimirovich, G\. Zubkova, P\. Blinov, and A\. Savchenko\(2025\)3mdbench: medical multimodal multi\-agent dialogue benchmark\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 26625–26665\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[44\]G\. Team, T\. Mesnard, C\. Hardin, R\. Dadashi, S\. Bhupatiraju, S\. Pathak, L\. Sifre, M\. Rivière, M\. S\. Kale, J\. Love,et al\.\(2024\)Gemma: open models based on gemini research and technology\.arXiv preprint arXiv:2403\.08295\.Cited by:[§3\.1](https://arxiv.org/html/2607.08257#S3.SS1.p1.2)\.
- \[45\]K\. Team, Y\. Bai, Y\. Bao, Y\. Charles, C\. Chen, G\. Chen, H\. Chen, H\. Chen, J\. Chen, N\. Chen,et al\.\(2025\)Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[§3\.1](https://arxiv.org/html/2607.08257#S3.SS1.p1.2)\.
- \[46\]T\. Tu, S\. Azizi, D\. Driess, M\. Schaekermann, M\. Amin, P\. Chang, A\. Carroll, C\. Lau, R\. Tanno, I\. Ktena,et al\.\(2024\)Towards generalist biomedical ai\.Nejm Ai1\(3\),pp\. AIoa2300138\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[47\]S\. Wang, Z\. Tang, H\. Yang, Q\. Gong, T\. Gu, H\. Ma, Y\. Wang, W\. Sun, Z\. Lian, K\. Mao,et al\.\(2025\)A novel evaluation benchmark for medical llms illuminating safety and effectiveness in clinical domains\.npj Digital Medicine\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[48\]G\. Williams, S\. Rutunda, F\. Nzabakira, and B\. A\. Mateen\(2025\)Human evaluators vs\. llm\-as\-a\-judge: toward scalable, real\-time evaluation of genai in global health\.medRxiv,pp\. 2025–10\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[49\]J\. Xu, T\. Wei, B\. Hou, P\. Orzechowski, S\. Yang, R\. Jin, R\. Paulbeck, J\. Wagenaar, G\. Demiris, and L\. Shen\(2025\)Mentalchat16k: a benchmark dataset for conversational mental health assistance\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5367–5378\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[50\]S\. Xu, T\. Zhou, J\. Ma, Y\. Ding, Y\. Yan, M\. Xiao, G\. Li, H\. Geng, Y\. Han, J\. Chen,et al\.\(2026\)LingxiDiagBench: a multi\-agent framework for benchmarking llms in chinese psychiatric consultation and diagnosis\.arXiv preprint arXiv:2602\.09379\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.6.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[51\]Z\. Yan, D\. Song, Z\. Fang, Y\. Ji, X\. Li, Q\. Li, and L\. Sun\(2026\)Livemedbench: a contamination\-free medical benchmark for llms with automated rubric evaluation\.arXiv preprint arXiv:2602\.10367\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[52\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4](https://arxiv.org/html/2607.08257#S4.SS0.SSS0.Px3.p1.1)\.
- \[53\]Z\. Yao, M\. Sun, W\. S\. Jang, S\. Kwon, S\. Kwon, and H\. Yu\(2025\)DischargeSim: a simulation benchmark for educational doctor–patient communication at discharge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10783–10809\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[54\]C\. Zhang, R\. Li, M\. Tan, M\. Yang, J\. Zhu, D\. Yang, J\. Zhao, G\. Ye, C\. Li, and X\. Hu\(2024\)Cpsycoun: a report\-based multi\-turn dialogue reconstruction and evaluation framework for chinese psychological counseling\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 13947–13966\.Cited by:[Table 1](https://arxiv.org/html/2607.08257#S1.T1.9.1.5.1),[§1](https://arxiv.org/html/2607.08257#S1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[55\]M\. Zhang, X\. Yang, X\. Zhang, T\. Labrum, J\. C\. Chiu, S\. M\. Eack, F\. Fang, W\. Y\. Wang, and Z\. Chen\(2025\)CBT\-bench: evaluating large language models on assisting cognitive behavior therapy\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 3864–3900\.Cited by:[§1](https://arxiv.org/html/2607.08257#S1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[56\]M\. Zhang, Y\. Shen, Z\. Li, H\. Sha, B\. Hu, Y\. Wang, C\. Huang, S\. Liu, J\. Tong, C\. Jiang,et al\.\(2025\)LLMEval\-med: a real\-world clinical benchmark for medical llms with physician validation\.arXiv preprint arXiv:2506\.04078\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[57\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2019\)Bertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.
- \[58\]X\. Zheng, H\. Li, W\. Luo, W\. Zhai, Y\. Li, C\. Yan, T\. Tang, Y\. Ma, K\. Yang, D\. Liu,et al\.\(2026\)ClinConsensus: a consensus\-based benchmark for evaluating chinese medical llms across difficulty levels\.arXiv preprint arXiv:2603\.02097\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px1.p1.1)\.
- \[59\]S\. Zhou, W\. Xie, J\. Li, Z\. Zhan, M\. Song, H\. Yang, C\. Espinoza, L\. Welton, X\. Mai, Y\. Jin,et al\.\(2025\)Automating expert\-level medical reasoning evaluation of large language models\.npj Digital Medicine\.Cited by:[§5](https://arxiv.org/html/2607.08257#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix ALimitations

MentalHospital currently focuses on text\-based psychiatric interaction\. Although we explored generative digital humans for richer patient presentation, current techniques remain insufficient for producing pathology\-level, clinically discriminative facial expressions, prosody, and subtle behaviors required for psychiatric assessment\. We therefore use text\-based simulation to preserve controllability, clinical grounding, and evaluation reliability\. In addition, MentalHospital does not yet provide systematic fairness, bias, or adversarial safety evaluation for scenarios such as self\-harm escalation, psychosis reinforcement, medication misuse, or demographic bias\. Future work will extend the benchmark toward clinically faithful multimodal simulation, safety\-oriented stress tests, and broader validation across clinical settings\.

## Appendix BEthics Statement

## Appendix CResource Release and Access Protocol

To support reproducibility while protecting clinical privacy, we adopt a tiered release protocol\. Publicly released resources will include the MentalHospital environment code, evaluation scripts, prompts, rubrics, metric implementations, model\-output logs, and the MentalEval evaluator weights with corresponding inference scripts\. These resources are sufficient to reproduce the evaluation pipeline and apply MentalEval to new model outputs\.

Raw EHRs, identifiable clinical records, case\-level clinical narratives, and MentalEval training data will not be publicly released due to clinical privacy constraints\. De\-identified benchmark cases and derived evaluation checkpoints will be provided only through controlled research access, subject to data\-use approval\. This protocol separates reproducible evaluation components from sensitive clinical source data and reduces the risk of privacy leakage\.

## Appendix DSafety, Bias, and Generalizability\.

MentalHospital is designed for controlled evaluation rather than direct clinical deployment\. Although the benchmark covers multi\-center psychiatric EHR cases across three regions, its standardized patient behaviors, clinical documentation patterns, and evaluator judgments may still reflect local practice conventions and expert preferences\. In addition, EHR\-derived references may encode historical diagnostic or examination biases\. MentalEval assesses process\-level quality, empathy, diagnostic rigor, and treatment appropriateness, but it does not fully cover all safety\-critical scenarios such as self\-harm escalation, psychosis reinforcement, medication misuse, or demographic bias\. Future work will extend MentalHospital with adversarial safety cases, fairness\-oriented evaluation, and culture\-sensitive patient simulation\.

## Appendix EData Distribution

In this section, we provide the full forms and brief explanations of the category abbreviations used in the main text\. Fig[8](https://arxiv.org/html/2607.08257#A5.F8)presents the names and corresponding codes for the 76 specific disorders included in the study\.

The full form ofANXis Anxiety or fear\-related disorders\. Anxiety or fear\-related disorders are a group of mental health conditions characterized by excessive fear and anxiety, along with associated behavioral disturbances\. These symptoms are severe enough to cause significant distress or impairment in an individual’s personal, social, educational, occupational, or other important areas of functioning\. According to the ICD\-11, specific types of disorders under this category include generalized anxiety disorder, panic disorder, agoraphobia, specific phobias, social anxiety disorder, and separation anxiety disorder\. Patients may exhibit specific cognitive features, such as excessive worry about particular situations or objects, which help distinguish between different types of anxiety or fear\-related disorders\.

The full form ofCATAis Catatonia\. Catatonia is a clinical syndrome characterized by psychomotor disturbances, which manifest as a combination of reduced, increased, or abnormal psychomotor activity\. Typical symptoms include stupor \(maintaining a fixed posture for extended periods\), rigidity \(abnormal increase in muscle tension\), waxy flexibility \(limbs can be manipulated and maintained in a posture\), mutism, negativism \(active or passive resistance to instructions\), echolalia \(repetition of others’ speech\), or echopraxia \(repetition of others’ actions\)\. This syndrome may arise secondary to psychiatric disorders such as schizophrenia, bipolar disorder, and depressive disorders, and can also be triggered by drug reactions, neurological diseases, or physical illnesses\.

![Refer to caption](https://arxiv.org/html/2607.08257v1/x6.png)Figure 6:Age distribution in MentalHospital\. Left: single\-diagnosis cases; right: comorbid cases\.The full form ofSUDis Disorders due to substance use or addictive behaviors\. These disorders refer to functional impairment or distress caused by the repeated use of psychoactive substances \(including drugs\) or specific repetitive rewarding behaviors, manifesting in cognitive, behavioral, and physiological symptoms\. The core features include a persistent craving for the substance or behavior, impaired control, prioritizing use despite harm, and neuroadaptive changes\. Key characteristics also include difficulty controlling the frequency, duration, and intensity of use, withdrawal symptoms after reducing or stopping use \(such as tremors, anxiety, insomnia\), tolerance \(requiring increased doses or intensity for the same effect\), functional impairment in personal, family, occupational, or social roles, and continued use despite knowing the harm to physical and mental health\.

The full form ofBODis Disorders of bodily distress or bodily experience\. These disorders are mental health conditions primarily characterized by bodily symptoms\. Patients often show excessive worry about their health, accompanied by persistent physical symptoms such as pain, fatigue, and digestive issues, among others\. These symptoms typically cannot be fully explained by routine medical examinations\.

The full form ofSTRESSis Disorders specifically associated with stress\. These disorders are a group of mental health conditions triggered by clearly identifiable stressors or traumatic events\. Patients typically present with acute stress disorder, which occurs within days of exposure to extreme stressors and is characterized by intense fear, helplessness, and flashbacks\. Post\-traumatic stress disorder \(PTSD\) lasts longer and is marked by the recurrent re\-experiencing of traumatic events, avoidance behaviors, and heightened arousal\. Adjustment disorder is another form, where emotional or behavioral problems arise from identifiable life events that exceed normal coping mechanisms, typically occurring within one month of the event and affecting daily functioning\.

The full form ofDISRis Disruptive behavior or dissocial disorders\. Impulse control disorders are a group of mental health conditions characterized by the difficulty of patients to resist strong, inappropriate desires or impulses, leading to the performance of certain specific behaviors\. These behaviors often cause long\-term harm to the individual or others, or result in significant impairment in important functional areas such as social, family, or occupational domains\. Although patients typically recognize the negative impact of these behaviors, they are unable to resist the impulse in the short term\. Common ICDs include pyromania, kleptomania, intermittent explosive disorder, and compulsive sexual behavior disorder\.

The full form ofDISSis Dissociative disorders\. Dissociative disorders are complex mental health conditions primarily characterized by the involuntary partial or complete disintegration of psychological functions such as identity, memory, consciousness, perception, or behavior\. These symptoms are not directly caused by drugs, substance abuse, or other psychiatric, behavioral, or neurodevelopmental disorders, and must be distinguished from normal manifestations in cultural or religious practices\. The main types include, but are not limited to, dissociative amnesia \(selective memory loss\), depersonalization\-derealization disorder \(feeling detached from reality\), dissociative fugue \(manifesting as a confused state\), and dissociative identity disorder \(formerly known as multiple personality disorder\)\.

The full form ofELIMis Elimination disorders\. Elimination disorders refer to the persistent or recurrent inability to control urination or defecation after reaching the normal developmental age\. The main manifestations include enuresis, which is the involuntary urination during the night or day, typically in bed or clothing, and encopresis, which is the inappropriate defecation in places such as clothing\. This diagnosis is applicable when children fail to achieve typical bladder and bowel control by the expected developmental age \(enuresis by age 5 and encopresis by age 4\)\. If the incontinence symptoms can be fully attributed to other health conditions, congenital or acquired abnormalities in the urinary or gastrointestinal systems, or medication side effects, it should not be diagnosed as an elimination disorder\.

The full form ofEATis Feeding or eating disorders\. Feeding or eating disorders are a group of mental health conditions characterized by abnormal eating behaviors that cannot be explained by other health conditions and do not conform to normal developmental patterns in the cultural context\. These include, but are not limited to, rumination/regurgitation, avoidant/restrictive food intake, anorexia nervosa, bulimia, binge eating, and pica\. These disorders often involve issues with controlling food intake, concerns about weight and body shape, or inappropriate dietary choices\.

The full form ofPREGis Mental or behavioral disorders associated with pregnancy, childbirth, or the puerperium\. These disorders refer to mental health issues that occur during pregnancy, during childbirth, and within the first six weeks postpartum\. Such disorders include, but are not limited to, depressive disorders, anxiety disorders, and psychotic disorders\. Based on their severity and specific manifestations, they can be further categorized into non\-psychotic disorders \(e\.g\., postpartum depression\) and psychotic disorders \(e\.g\., postpartum psychosis\)\. These mental health conditions significantly affect a woman’s psychological state and social functioning, requiring professional medical intervention\.

![Refer to caption](https://arxiv.org/html/2607.08257v1/x7.png)Figure 7:Disorder distribution\. Left: single\-diagnosis cases; right: comorbid cases\.The full form ofMOODis Mood disorders\. Mood disorders are a group of mental health conditions characterized by significant and persistent changes in mood or emotional states\. These disorders include two main types: bipolar disorder and depressive disorder\. They are typically manifested by several major types of mood episodes: depressive episodes, which involve persistent low mood, loss of interest, and reduced energy; manic episodes, which involve abnormally elevated mood, increased activity, and reduced need for sleep; mixed episodes, where both depressive and manic symptoms are present simultaneously; and hypomanic episodes, which are similar to manic episodes but less severe, with minimal impairment of social functioning\. These mood episodes are not independent diagnostic entities but are the primary components of mood disorders\.

The full form ofNCOGis Neurocognitive disorders\. Neurocognitive disorders refer to major clinical impairments in cognitive function that are acquired, rather than developmental or congenital\. Specifically, these disorders do not include cognitive issues that are present from birth, nor do they cover problems that typically emerge during the developmental stage \(which are classified as neurodevelopmental disorders\)\. Instead, neurocognitive disorders refer to a decline in cognitive function from a previously attained level\.

The full form ofNDEVis Neurodevelopmental disorders\. Neurodevelopmental disorders are a group of mental or behavioral disorders that appear during an individual’s developmental stage, typically in early childhood, especially before school age, and are characterized by developmental deficits that cause impairment in social, academic, or occupational functioning\. The core features of these disorders include significant difficulties in acquiring and performing specific intellectual, motor, language, or social functions\. While many mental and behavioral disorders can emerge during the developmental period, only those disorders that are primarily characterized by neurodevelopmental impairments are classified under this group\.

The full form ofOCDis Obsessive\-compulsive or related disorders\. Obsessive\-compulsive or related disorders are a group of mental health conditions characterized by repetitive thoughts and behaviors\. The main symptoms include: obsessive thoughts, which are unwanted, recurring ideas, doubts, or impulses, and compulsive behaviors, which are repetitive actions or mental rituals\. These symptoms persist despite the patient’s awareness of their meaninglessness, leading to significant personal distress and impairment in social functioning\.

The full form ofPERSis Personality disorders and related traits\. Personality disorders refer to a persistent and stable pattern of emotional experiences, cognitive patterns, and behaviors that significantly deviate from the cultural expectations of the individual’s background, leading to widespread functional impairment or subjective distress\. This pattern typically emerges during adolescence or early adulthood, with its stability confirmed through observation in adulthood\. Personality disorders and related personality traits cover two main aspects: prominent personality features or patterns, where individuals exhibit significant deviations in personality traits, but the severity or extent may not fully meet the diagnostic criteria for a personality disorder, and personality disorders, which include various subtypes \(such as borderline, antisocial, schizoid, etc\.\), characterized by persistent functional impairment, and must exclude direct consequences of other psychiatric disorders or substance use\.

The full form ofSCHIZis Schizophrenia or other primary psychotic disorders\. Schizophrenia and other primary psychotic disorders are a group of mental health conditions characterized by significant impairment in reality testing\. The core symptoms include: positive symptoms such as delusions, hallucinations, and disorganized speech; negative symptoms such as flat affect, reduced volition, and social withdrawal; and cognitive dysfunctions such as impairments in attention, memory, and executive function\. These symptoms cause the individual’s thoughts, emotions, and behaviors to deviate markedly from the cultural norms, and cannot be attributed to other psychiatric disorders, such as bipolar disorder or substance use disorders\.

Specific Disorders List\["6A20 Schizophrenia","6B00 Generalised anxiety disorder","6A72 Dysthymic disorder","6E20 Mental or behavioural disorders associated with pregnancy, childbirth or the puerperium, without psychotic symptoms","6A61 Bipolar type II disorder","6A00 Disorders of intellectual development","6D70 Delirium","6E21 Mental or behavioural disorders associated with pregnancy, childbirth or the puerperium, with psychotic symptoms","6B20 Obsessive\-compulsive disorder","6A40 Catatonia associated with another mental disorder","6D72 Amnestic disorder","6A62 Cyclothymic disorder","6A23 Acute and transient psychotic disorder","6B43 Adjustment disorder","6A21 Schizoaffective disorder","6A70 Single episode depressive disorder","6A73 Mixed depressive and anxiety disorder","6A71 Recurrent depressive disorder","6A24 Delusional disorder","6A60 Bipolar type I disorder","6C20 Bodily distress disorder","6A22 Schizotypal disorder","6A41 Catatonia induced by substances or medications","6D71 Mild neurocognitive disorder","6D85 Dementia due to diseases classified elsewhere","6C4G Disorders due to use of unknown psychoactive substances","6C4E Disorders due to use of other specified psychoactive substances, including medications","6C46 Disorders due to use of stimulants including amphetamines, methamphetamine or methcathinone","6B04 Social anxiety disorder","6C40 Disorders due to use of alcohol","6D10 Personality disorder","6A02 Autism spectrum disorder","6B02 Agoraphobia","6B60 Dissociative neurological symptom disorder","6C91 Conduct\-dissocial disorder","6C90 Oppositional defiant disorder","6B83 Avoidant\-restrictive food intake disorder","6B40 Post traumatic stress disorder","6C4F Disorders due to use of multiple specified psychoactive substances, including medications","6D84 Dementia due to psychoactive substances including medications","6B63 Possession trance disorder","6B03 Specific phobia","6B41 Complex post traumatic stress disorder","6A03 Developmental learning disorder","6C51 Gaming disorder","6D86 Behavioural or psychological disturbances in dementia","6A06 Stereotyped movement disorder","6D83 Frontotemporal dementia","6B80 Anorexia Nervosa","6B23 Hypochondriasis","6C49 Disorders due to use of hallucinogens","6B81 Bulimia Nervosa","6A05 Attention deficit hyperactivity disorder","6B62 Trance disorder","6B84 Pica","6B21 Body dysmorphic disorder","6B05 Separation anxiety disorder","6B64 Dissociative identity disorder","6B61 Dissociative amnesia","6A04 Developmental motor coordination disorder","6C4D Disorders due to use of dissociative drugs including ketamine and phencyclidine \[PCP\]","6B66 Depersonalization\-derealization disorder","6B25 Body\-focused repetitive behaviour disorders","6B42 Prolonged grief disorder","6B82 Binge eating disorder","6C44 Disorders due to use of sedatives, hypnotics or anxiolytics","6B22 Olfactory reference disorder","6C01 Encopresis","6D81 Dementia due to cerebrovascular disease","6B06 Selective mutism","6C50 Gambling disorder","6B24 Hoarding disorder","6C41 Disorders due to use of cannabis","6B85 Rumination\-regurgitation disorder","6C47 Disorders due to use of synthetic cathinones"\]Figure 8:Specific Disorders List\.A list of 76 disorders mentioned in the paper, including their corresponding codes and specific names\.
## Appendix FData Processing and Ethical Compliance

##### Data Source and Case Composition\.

MentalHospital is constructed from 1,193 de\-identified psychiatric electronic health record \(EHR\) cases collected from multiple collaborating clinical centers\. The dataset contains 885 single\-diagnosis cases and 308 comorbid cases, covering both typical psychiatric presentations and clinically complex comorbidity patterns\. The age distribution of single\-diagnosis and comorbid cases is shown in Figure[6](https://arxiv.org/html/2607.08257#A5.F6), and the corresponding disorder\-level distribution is summarized in Figure[7](https://arxiv.org/html/2607.08257#A5.F7)\. For each casecc, two licensed psychiatrists independently reviewed the EHR to ensure that the record was clinically complete and that the final diagnosis was consistent with the ICD\-11 classification of mental, behavioral, and neurodevelopmental disorders\. Cases with incomplete clinical information, ambiguous diagnostic conclusions, or unresolved annotation conflicts were excluded from benchmark construction\. Disagreements between the two reviewers were resolved through expert consensus\.

##### EHR Content and Case Structure\.

Each finalized EHR contains three types of clinical information\. To standardize these fields, we use a structured extraction prompt to convert unstructured Chinese psychiatric records into a fixed JSON schema, as shown in Figure[9](https://arxiv.org/html/2607.08257#A6.F9)\. Figures[10](https://arxiv.org/html/2607.08257#A6.F10)and[11](https://arxiv.org/html/2607.08257#A6.F11)provide an example of the original psychiatric clinical record and its corresponding structured version\. The structured record consists of four major components: basic demographic information, clinical formulation, diagnostic category, and disorder\-level diagnosis\. The clinical formulation is further organized into several evidence fields, including chief complaint and present illness, physical examination, mental status examination, past history, family history, auxiliary examination results, and medication history\. This structure separates patient\-side evidence, examination\-side evidence, and diagnostic labels, thereby enabling consistent case instantiation and evaluation\.

Specifically, patient\-side information includes chief complaint, present illness, past history, family history, personal history, medication history, and mental status examination\. Examination information includes physical examination findings and auxiliary examination results\. Diagnostic information includes the final diagnostic category and disorder\-level diagnosis\. These fields are used to instantiate the formal case representation in MentalHospital\. For each casecc, patient\-side information is converted into𝒦cpat\\mathcal\{K\}^\{\\mathrm\{pat\}\}\_\{c\}, examination information is converted into𝒦cexam\\mathcal\{K\}^\{\\mathrm\{exam\}\}\_\{c\}, and diagnostic information is used to construct the reference diagnostic target\. For comorbid cases, all clinically confirmed diagnoses are retained and represented in the disorder\-level diagnostic target\.

Psychiatric Clinical Record Structuring PromptYou are an assistant for structured information extraction from psychiatric clinical records\. Please convert the input unstructured Chinese clinical record into a fixed JSON format\.The output format must strictly follow:```
{
  "id": "",
  "basiclnfomation": "",
  "realFormulation": {
    "chiefComplaint": [],
    "physicalExamination": [],
    "mentalStatus": [],
    "pastHistory": [],
    "familyHistory": [],
    "auxiliaryExamination": {},
    "medicationHistory": []
  },
  "realCategory": [],
  "realDisorder": []
}
```

The extraction rules are as follows:•id: Directly copy the inputid\.•basiclnfomation: Retain only the most basic demographic information, and by default write it as “Patient X, X years old\.”
…The constraints are as follows:1\.Strictly follow the original text\. Do not add, infer, or speculate\.2\.Retain negative information\.3\.Keep the wording as close to the original text as possible, with only necessary standardization\.4\.Remove duplicate array items, but do not merge independent factual units\.5\.Missing fields must be output as empty arrays, empty objects, or empty strings\.6\.Output only valid JSON\. Do not output explanations or Markdown\.7\.Do not output‘‘‘json,‘‘‘, or any other code\-fence markers\.Input:```
<INPUT_JSON>
```

Figure 9:System prompt for structuring psychiatric clinical records into standardized JSON format\.Original Medical RecordBasic information:Male, 24\-year\-old\.Chief complaint:The main clinical manifestations were as follows: more than two years ago, while attending university, the patient had conflicts with classmates and subsequently developed fear without apparent reason, became unwilling to study, felt that someone was monitoring him and intended to harm him, and showed decreased interest\. …Past history and family history:the patient was previously healthy…Physical examination:T 36\.6∘C, P 96 beats/min, R 20 breaths/min, BP 118/76 mmHg…Mental status examination:consciousness was clear, orientation was intact, attention was impaired, responses were irrelevant to the questions, and the voice was low\. The patient was appropriately dressed, had an expressionless facial expression, was partially independent in daily living, had passive contact with the surroundings, and was indifferent to medical staff and family members…Medication history:On January x, 20xx, liver function, renal function, electrolytes, and complete blood count in our hospital were normal, and urinalysis showed no obvious abnormality\. Chest X\-ray, electrocardiogram, and electroencephalography showed no obvious abnormality\.Figure 10:Example of a original clinical record in MentalHospital\.Structured Medical Record```
{
  "id": "1000769",
  "basicInfomation": "Male, 24-year-old",
  "realFormulation": {
    "chiefComplaint": [
      "More than 2 years ago, the patient had conflicts with classmates",
      "More than 2 years ago, the patient developed fear without apparent reason",
      "More than 2 years ago, the patient became unwilling to study",
       ......
    ],
    "physicalExamination": [
      "T 36.6°C",
      "P 96 beats/min",
      "R 20 breaths/min",
       ......
      "Physiological reflexes were present",
      "Pathological reflexes were not elicited"
    ],
    "mentalStatus": [
      "Consciousness was clear",
      "Orientation was normal",
       ......
    ],
    "pastHistory": [
      "Previously healthy",
       ......
    ],
    "familyHistory": [
       ......
    ],
    "auxiliaryExamination": {
       ......
    },
    "medicationHistory": [
       ......
    ]
  },
  "realCategory": [
    "Schizophrenia or other primary psychotic disorders"
  ],
  "realDisorder": [
    "6A20 Schizophrenia"
  ]
}
```

Figure 11:Example of a structured psychiatric clinical record in MentalHospital\.
##### Standardized De\-identification Pipeline\.

Each raw EHR was converted into a benchmark case through a standardized on\-site processing pipeline\. All automated processing was conducted within secure local environments of the participating clinical centers\. We used an on\-premise DeepSeek\-R1\-assisted pipeline\[[27](https://arxiv.org/html/2607.08257#bib.bib44)\], together with rule\-based detectors and manual expert review, to remove or generalize potentially identifiable information from the raw clinical records\. Sensitive elements such as exact dates, locations, institution names, contact information, and other personally identifying details were either masked or replaced with generalized clinical descriptions\. For example, exact calendar dates were converted into relative time expressions or clinically meaningful time ranges, and specific locations or institutions were replaced with generalized contextual descriptions when needed for clinical coherence\.

##### Privacy Audit\.

To verify the effectiveness of de\-identification, we conducted a multi\-stage privacy audit\. The automated audit first scanned all processed records for residual sensitive information\. As shown in Table[6](https://arxiv.org/html/2607.08257#A6.T6), the pipeline detected 623 date mentions, 375 location mentions, and 525 institution mentions, all of which were removed, masked, or generalized during de\-identification\. Additional categories of potentially identifiable information were also checked using rule\-based matching and model\-assisted screening\. After automated processing, licensed psychiatrists manually reviewed the processed records to verify that no directly identifiable patient information remained and that the rewritten clinical content preserved the original psychopathological logic\. In the final manual audit, no residual directly identifiable information was found\.

Table 6:Privacy Audit ResultsEntity CategoryPresidio FlagsAnonymizedBreachesPatient/Physician Name0–0Specific Date623623 \(100%\)0Detailed Location375375 \(100%\)0Organization/Workplace525525 \(100%\)0
##### Clinical Validity Review\.

Because de\-identification can potentially alter clinical meaning, all processed records were further reviewed for clinical validity\. Licensed psychiatrists checked whether the transformed records preserved the original symptom structure, temporal course, mental status findings, examination results, and diagnostic rationale\. When a sensitive detail was clinically relevant, it was replaced with a generalized but clinically equivalent description rather than being removed outright\. This ensured that the benchmark cases remained clinically coherent while satisfying privacy requirements\.

##### Checkpoint Construction\.

To support fine\-grained objective evaluation, we converted finalized EHR fields into structured checkpoint arrays\. We used the on\-premise DeepSeek\-R1 model\[[27](https://arxiv.org/html/2607.08257#bib.bib44)\]to extract checkpoints from the de\-identified EHR text\. The extracted checkpoints cover final diagnostic categories, disorder\-level diagnoses, chief complaints, present illness, past history, family history, personal history, physical examination findings, auxiliary examination findings, mental status examination observations, and treatment\-related information\. These checkpoints were represented as structured arrays so that different stages of the MentalHospital evaluation could be matched against case\-level references\.

##### Checkpoint Validation\.

All automatically extracted checkpoints were validated by licensed psychiatrists\. The validation process checked whether each checkpoint was faithful to the de\-identified EHR, clinically meaningful, and suitable for objective evaluation\. Incorrect, redundant, ambiguous, or overly specific checkpoints were removed or rewritten\. The finalized checkpoint arrays serve as case\-level references for evaluating evidence recovery, examination recommendation matching, clinical\-note completeness, factual consistency, diagnostic coverage, diagnostic prioritization, and treatment appropriateness\. This design allows MentalHospital to evaluate not only final diagnostic correctness but also whether the doctor agent recovers and uses clinically relevant evidence throughout the interaction process\.

##### Ethical Approval and Consent Waiver\.

The study protocol, data processing pipeline, and annotation procedures were reviewed and approved by the institutional ethics boards of all participating clinical centers\. To preserve anonymity during double\-blind review, the names of the clinical centers and approval numbers are omitted in the submission version and will be disclosed upon acceptance\. All data collection, processing, and annotation procedures followed the ethical principles of theDeclaration of Helsinki\. Since all records were fully de\-identified before benchmark construction and no intervention was performed on patients, the requirement for individual informed consent was waived by the relevant ethics boards\.

##### Data Security and Governance\.

No identifiable patient information was transferred outside the participating clinical centers\. All automated processing, including model\-assisted de\-identification and checkpoint extraction, was conducted using on\-premise deployments within secure hospital environments\. For de\-identification, we used a dedicated anonymization prompt, as shown in Figure[12](https://arxiv.org/html/2607.08257#A6.F12), which removes identity\-identifying information while preserving clinically relevant facts and the original record format\. Intermediate raw records and processing artifacts were retained locally under institutional data governance\. Only de\-identified benchmark cases and validated structured references were used for research analysis and model evaluation\.

Anonymization PromptAnonymizationTask DescriptionGiven an admission record, anonymize the identity\-related private information contained in the text\.Rules•Only anonymize the following content: patient names, physician names, hospital names, school names, real geographic locations, detailed addresses, specific institution names, and other information that can directly identify an individual or institution\.•Replace all anonymized content uniformly with\*\*\*\*\*\*\.•Clinical information, including age, sex, time, symptoms, diagnoses, examination results, medical history, behavioral descriptions, and related clinical facts, must be retained and must not be anonymized\.•“Sensitive” does not necessarily mean “private\.” Only identity\-identifying information should be processed; clinical factual information should not be modified\.Requirements•Perform anonymization only\. Do not add, delete, rewrite, summarize, or polish any other content\.•Strictly preserve the original format\.•Output the result as a string\.•Output only the processed result\. Do not output explanations or code fences, including‘‘‘plaintextor‘‘‘\.Input```
{INPUT}
```

OutputFigure 12:System prompt for privacy\-preserving anonymization of psychiatric admission records\.

## Appendix GInteraction Prompt

To instantiate the sequential psychiatric workflow in MentalHospital, we design a set of stage\-specific system prompts that regulate the doctor agent’s behavior, available information, output format, and transition condition at each step\. These prompts serve as the operational interface between the doctor agent and the case\-conditioned environmentℰc\\mathcal\{E\}\_\{c\}, ensuring that the agent completes the clinical process in a controlled and clinically aligned manner\.

Figure[13](https://arxiv.org/html/2607.08257#A7.F13)presents the prompt for theClinical History Collectionstage\. In this stage, the doctor agent conducts a multi\-turn psychiatric interview with the standardized patient𝒫c\\mathcal\{P\}\_\{c\}\. The prompt restricts the agent to history taking before the termination marker is produced, encouraging the agent to actively elicit patient\-side evidence, including chief complaints, symptom evolution, illness history, psychosocial context, and mental status information\.

Figure[14](https://arxiv.org/html/2607.08257#A7.F14)shows the prompt for theExamination Recommendationstage\. After the interview is completed, the doctor agent recommends a set of clinically necessary auxiliary examinations\. The prompt requires the output to be a JSON array enclosed by explicit markers, and each examination item must correspond to a specific and actionable clinical test\. This design avoids overly broad recommendations and enables direct comparison with the EHR\-derived examination\-side evidence𝒦cexam\\mathcal\{K\}^\{\\mathrm\{exam\}\}\_\{c\}\.

Figure[15](https://arxiv.org/html/2607.08257#A7.F15)illustrates the prompt for theClinical Formulationstage\. Given the accumulated dialogue context and the returned examination results, the doctor agent generates a concise clinical record\. The prompt emphasizes plain\-text clinical documentation and requires the agent to integrate patient information, interview findings, and auxiliary examination results into a coherent case formulation\.

Figure[16](https://arxiv.org/html/2607.08257#A7.F16)presents the prompt for theDefinitive Diagnosisstage\. In this stage, the doctor agent analyzes the complete clinical record and selects one or more final diagnoses from a predefined ICD\-based candidate list\. The prompt requires the agent to first provide diagnostic reasoning and supporting evidence, and then output the final diagnosis list using the exact candidate names\. This constraint supports standardized diagnostic evaluation and prevents uncontrolled free\-form diagnosis generation\.

Figure[17](https://arxiv.org/html/2607.08257#A7.F17)shows the prompt for theTreatment Planningstage\. Based on the complete clinical record, established diagnosis, and current clinical condition, the doctor agent generates a treatment plan enclosed by explicit output markers\. This final stage completes the S\.O\.A\.P\.\-style workflow and produces the treatment recommendationp^c\\hat\{p\}\_\{c\}for subsequent evaluation\.

Clinical History CollectionThe current stage is the first phase of the psychiatric clinical workflow:Clinical History Collection\.In this stage, you are required to conduct a multi\-turn conversation with the patient to collect clinical history information\.The available information is as follows:•Patient basic information:<BASIC\_INFO\>•Patient physical examination results:<PHYSICAL\_EXAMINATION\>Please begin the conversation with the patient directly and remain in this stage until you determine that sufficient clinical history information has been collected\.When you decide to conclude this stage, output the following marker alone:\[END\_INTERVIEW\]Before outputting\[END\_INTERVIEW\], your task remains clinical history collection\. Only after this marker is output will the workflow proceed to the next stage\.Figure 13:System prompt for the Clinical History Collection stage in the psychiatric clinical workflow\.Examination RecommendationThe current stage is the second phase of the psychiatric clinical workflow:Examination Recommendation\.In this stage, you are required to independently recommend the next auxiliary examinations based on the current dialogue context, the patient’s previous responses, and the clinical information already obtained\.Your task is to output a proposed list of auxiliary examinations to support subsequent diagnostic judgment and differential diagnosis\.The output requirements are as follows:1\.You should independently generate the auxiliary examinations that need to be requested based on the current case information\.2\.If you determine that no auxiliary examination is currently necessary, output an empty array:\[\]\.3\.Each item you output must be a specific, standardized, and clearly defined examination name, preferably using commonly accepted clinical medical terminology\.4\.The examination items you output must be sufficiently specific so that each item can be clearly mapped to an actual clinical examination\. Do not output overly broad, vague, or non\-actionable category names, such as imaging examination,” laboratory examination,” or blood test\.”5\.Only output auxiliary examinations that are genuinely necessary for the current diagnostic process, and avoid irrelevant, redundant, or clearly duplicated items\.6\.The final result must be enclosed by\[BEGIN\_EXAMINATIONS\]and\[END\_EXAMINATIONS\]\.7\.The content between\[BEGIN\_EXAMINATIONS\]and\[END\_EXAMINATIONS\]must be, and must only be, a JSON array\. Do not add any explanations, comments, or other content\.Example output:```
[BEGIN_EXAMINATIONS]
[
  "Complete blood count",
  "Thyroid function tests",
  "Brain MRI",
  "Electroencephalography"
]
[END_EXAMINATIONS]
```

If no auxiliary examination is currently required, output:```
[BEGIN_EXAMINATIONS]
[]
[END_EXAMINATIONS]
```

After you output\[END\_EXAMINATIONS\], the workflow will proceed to the next stage\.Figure 14:System prompt for auxiliary examination recommendation in the psychiatric clinical workflow\.Clinical FormulationBased on the auxiliary examination recommendations you proposed in the previous stage, the following examination results are now returned:<EXAMINATIONS\_RESULTS\>Notes:•If the result of an examination item isNONE, it indicates that no positive finding is provided for that examination\. This may mean that the examination was not performed, or that no obvious abnormality was found\.•Please complete the task for this stage based on the current dialogue context, the available patient information, and the examination results above\.The current stage is the third phase of the psychiatric clinical workflow:Clinical Formulation\.In this stage, you are required to organize and generate a clinical record based on the current dialogue context, together with the patient information and auxiliary examination results\.The output requirements are as follows:1\.Your output must be the main body of a clinical record\.2\.The clinical record must be plain text\. Do not output JSON, Markdown, explanatory notes, or any other additional content\.3\.You must enclose the entire clinical record using the following markers:```
[BEGIN_CLINICAL_FORMULATION]
Main body of the clinical record
[END_CLINICAL_FORMULATION]
```

4\.The clinical record should be concise, clear, and complete\. Avoid verbose descriptions and irrelevant elaboration while preserving all key information\.After you output\[END\_CLINICAL\_FORMULATION\], the workflow will proceed to the next stage\.Figure 15:System prompt for clinical record generation in the psychiatric clinical workflow\.Definitive DiagnosisThe current stage is the fourth phase of the psychiatric clinical workflow:Definitive Diagnosis\.In this stage, you are required to analyze the complete clinical record currently available, including the dialogue, examination results, and clinical formulation, and then select the final diagnosis or diagnoses that are most consistent with the clinical evidence from the given list of candidate diagnoses\.The candidate diagnoses are as follows:```
[
  "6A20 Schizophrenia",
  "6B00 Generalised anxiety disorder",
  "6A72 Dysthymic disorder",
   ......
  "6A41 Catatonia induced by substances or medications",
  "6D71 Mild neurocognitive disorder",
  "6D85 Dementia due to diseases classified elsewhere",
  "6C4G Disorders due to use of unknown psychoactive substances",
  "6C41 Disorders due to use of cannabis",
  "6B85 Rumination-regurgitation disorder",
  "6C47 Disorders due to use of synthetic cathinones"
]
```

The output requirements are as follows:1\.You must first provide the diagnostic rationale and supporting evidence to explain your diagnostic decision\-making process\.2\.After completing the diagnostic rationale, you must output the final diagnostic results in list format\.3\.You may select only one or more diagnoses from the candidate diagnosis list above, including comorbid diagnoses where applicable\.4\.The output order should prioritize the clinically primary diagnosis\.5\.The diagnosis names in the output array must exactly match the original text in the candidate diagnosis list\. They must not be rewritten, abbreviated, or merged\.6\.The final result must be enclosed by\[BEGIN\_DEFINITIVE\_DIAGNOSIS\]and\[END\_DEFINITIVE\_DIAGNOSIS\]\.7\.The content between\[BEGIN\_DEFINITIVE\_DIAGNOSIS\]and\[END\_DEFINITIVE\_DIAGNOSIS\]must be, and must only be, a JSON array\.Example output:```
...
[BEGIN_DEFINITIVE_DIAGNOSIS]
[
    "6A20 Schizophrenia",
    "6B21 Body dysmorphic disorder",
    "6C50 Gambling disorder"
]
[END_DEFINITIVE_DIAGNOSIS]
```

After you output\[END\_DEFINITIVE\_DIAGNOSIS\], the workflow will proceed to the next stage\.Figure 16:System prompt for evidence\-based definitive diagnosis in the psychiatric clinical workflow\.Treatment PlanningThe current stage is the fifth phase of the psychiatric clinical workflow:Treatment Planning\.In this stage, you are required to generate corresponding treatment recommendations based on the complete clinical record currently available, the established diagnostic results, and the current clinical condition\.Output Requirements:The complete treatment recommendations must be enclosed by\[BEGIN\_TREATMENT\_PLAN\]and\[END\_TREATMENT\_PLAN\]\.After you output\[END\_TREATMENT\_PLAN\], the workflow will be concluded\.Figure 17:System prompt for treatment planning in the psychiatric clinical workflow\.
## Appendix HEnvironment Modules

![Refer to caption](https://arxiv.org/html/2607.08257v1/x8.png)Figure 18:Overview of standardized patient construction in MentalHospital\. De\-identified psychiatric EHRs are transformed into role information, presentation skills, and memory skills to support clinically grounded patient simulation\.##### Prompt\-Level Patient Control\.

As shown in Figure[18](https://arxiv.org/html/2607.08257#A8.F18), the standardized patient is instantiated through a structured patient prompt that specifies the role boundary, available patient\-side information, response constraints, retrieval protocol, and output format\. As shown in Figure[19](https://arxiv.org/html/2607.08257#A8.F19), the patient agent is explicitly instructed to answer only as the patient, to avoid switching into the role of doctor, evaluator, observer, or record writer, and to refrain from fabricating unsupported facts, examinations, diagnoses, or treatment histories\. The prompt separates three types of information: basic demographic and role information, chief\-complaint evidence, and mental\-status presentation\. This separation ensures that role information is used only for patient identity setting, chief\-complaint entries serve as the factual basis of the patient’s subjective report, and mental\-status entries control observable behavior, affect, tone, and interaction style\.

##### Grounded Response Generation\.

To make patient simulation auditable, each response is required to include both the generated utterance and its grounding evidence in a structured JSON object\. Thepresentationfield captures brief observable behaviors, such as affect, hesitation, defensiveness, or reduced spontaneity, while theutterancefield contains the patient’s verbal reply\. The grounding fields record which original chief\-complaint and mental\-status entries are explicitly used in the current response\. Importantly, grounding entries must be copied exactly from the input rather than rewritten or summarized\. This design allows each patient response to be traced back to specific patient\-side evidence, reducing unsupported disclosure and preventing the model from introducing future\-stage clinical information that has not been asked about\.

##### Skill\-Guided Presentation and Retrieval\.

The patient skill further specifies how the patient should present and when memory retrieval should be activated\. Figure[20](https://arxiv.org/html/2607.08257#A8.F20)gives an example in which the patient’s mental status defines a persistently depressed affect, reduced social engagement, intact orientation, cooperative but defensive communication, and impaired social functioning\. These descriptions are not treated as facts to be directly recited in every turn; rather, they guide the style and behavioral presentation of the response\. The same skill file also defines a list of retrievable chief\-complaint events and their retrieval triggers\. When the doctor asks about specific experiential details, such as what happened, when the symptom started, how severe it was, how frequently it occurred, or what the situation was like, the patient first activates the corresponding retrieval command\. For simple greetings, broad questions, or already answered content, retrieval is not required\.

##### Topic\-Level Autobiographical Memory\.

The retrievable event memories provide detailed, topic\-level autobiographical evidence for the standardized patient\. As illustrated in Figure[21](https://arxiv.org/html/2607.08257#A8.F21), each memory unit corresponds to a clinically relevant event, such as depressed mood, poor sleep, reduced energy, decreased interest, unwillingness to go out, or unwillingness to work\. These memories transform EHR\-derived patient\-side evidence into first\-person experiential narratives, enabling the patient agent to provide concrete answers when the doctor asks detailed follow\-up questions\. Instead of exposing all clinical evidence at once, the patient discloses information gradually according to the current interview topic and retrieval state\. This supports a more realistic psychiatric interview process, where clinically important information must be elicited through targeted questioning\.

##### Interaction Workflow\.

During the interaction, the doctor agent’s question determines whether the patient should respond from the currently available context or retrieve a specific memory unit\. If no detailed retrieval is needed, the patient generates a brief, natural, and colloquial answer based on the active role information, chief complaint, mental status, and dialogue history\. If the question requires event\-specific details, the patient activates the relevant memory entry and then answers using only the retrieved content and the permitted context\. The final response therefore combines three components: a clinically grounded verbal reply, an optional observable presentation, and an explicit record of the evidence used\. This mechanism implements the skill\-augmented patient configuration described above, where the prompt enforces role and evidence constraints, the skill controls presentation and retrieval behavior, and the memory cases provide bounded experiential content for faithful patient simulation\.

##### Faithfulness and Clinical Utility\.

This design is intended to balance realism and controllability\. On the one hand, the patient can express symptoms in a natural first\-person manner, with affective and behavioral variation guided by the mental\-status skill\. On the other hand, the patient is constrained to disclose only information supported by the provided evidence or retrieved memory\. As a result, the simulated interview remains clinically meaningful while preserving evidence faithfulness\. The doctor agent must actively elicit symptom details, illness course, functional impairment, and related experiences, rather than receiving a complete case summary at the beginning of the interaction\. This makes the standardized patient suitable for evaluating whether doctor agents can conduct structured psychiatric interviews, recover patient\-side evidence, and progress toward clinically grounded assessment\.

##### Prompt\-Level Examination Retrieval\.

The concrete prompt used by the examination module is shown in Figure[22](https://arxiv.org/html/2607.08257#A8.F22)\. The auxiliary examination results used by this module are not generated freely by the model, but are obtained from the structured JSON medical record constructed during case preprocessing\. Specifically, after the original EHR is converted into structured patient\-side and examination\-side evidence, all available auxiliary examinations and their corresponding reports are stored in𝒦cexam\\mathcal\{K\}^\{\\mathrm\{exam\}\}\_\{c\}as standardized key–value entries\. These entries are then injected into the examination prompt through the<AUXILIARY\_EXAMINATIONS\>field, while the doctor agent’s requested examination array is injected through<EXAMINATION\_QUERY\>\.

The examination module therefore acts as a constrained retrieval interface over the structured JSON record\. Given a set of requested examinations, the module returns only the results that can be matched to existing entries in𝒦cexam\\mathcal\{K\}^\{\\mathrm\{exam\}\}\_\{c\}\. Because the doctor agent may use non\-standard or semantically related examination names, the prompt permits cautious semantic matching rather than requiring exact string matching\. However, this matching is deliberately strict: a result can be returned only when the requested item can be clearly mapped to one known examination item in the structured record\. If no clear mapping exists, the module returnsNONEfor that requested item\.

The output format is also constrained to prevent information leakage\. As specified in Figure[22](https://arxiv.org/html/2607.08257#A8.F22), the module must output a valid JSON object, preserve the original examination names requested by the doctor agent as keys, and maintain one\-to\-one correspondence between requested items and returned values\. It is not allowed to merge multiple examination results, return unrequested items, add explanatory comments, infer diagnoses, or provide treatment recommendations\. This ensures that the doctor agent can obtain examination evidence only through explicit requests, rather than receiving the full examination\-side record at once\.

This design keeps the hospital\-side examination process both clinically grounded and auditable\. Since all returned reports come from the preprocessed structured JSON case record, the module remains faithful to the available EHR\-derived evidence\. At the same time, each request made by the doctor agent is recorded as part of𝒜^cexam\\hat\{\\mathcal\{A\}\}^\{\\mathrm\{exam\}\}\_\{c\}, enabling objective comparison with the EHR\-derived reference examination set\. Thus, the examination module supports two functions simultaneously: it provides controlled access to auxiliary examination results during the simulated clinical episode, and it produces a traceable record of the examinations requested by the doctor agent for later coverage evaluation\.

Patient PromptYou are a standardized psychiatric patient being interviewed by a doctor\.Available information:•BASIC INFO:<BASIC\_INFO\>•CHIEF COMPLAINT:<CHIEF\_COMPLAINT\>•MENTAL STATUS:<MENTAL\_STATUS\>Your task is to answer the doctor’s questions as the patient, based only on the given information and dialogue context\.Rules:1\.Always respond as the patient; do not switch to the role of doctor, evaluator, observer, or record writer\.2\.Do not fabricate facts, examinations, diagnoses, or treatment history\.3\.BASIC INFOis only for role setting and is not included in grounding\.CHIEF COMPLAINTis the main factual basis\.MENTAL STATUScontrols observable presentation, tone, affect, and interaction style\.
……
Retrieval is optional and should be used only for detailed follow\-up questions, such as requests about specific experiences, timing, frequency, severity, or concrete scenarios\. It is not needed for greetings, simple questions, or questions already answered\.Examples:```
[SEARCH_poor sleep at night]
[SEARCH_feeling followed]
[SEARCH_suspiciousness]
```

The available event list is provided inskill\.md\.Response Requirements:You must answer the doctor’s question regardless of whether retrieval is used\. Output only one JSON object:```
{
  "presentation": "",
  "utterance": "",
  "grounding": {
    "chief_complaint": [],
    "mental_status": []
  }
}
```

1\.presentation: brief observable behavior in parentheses; leave empty if absent\. ……For role setting, refer to:```
<PATIENT_SKILL>
```

Now output the patient’s next response according to the doctor’s question\.Figure 19:System prompt for standardized psychiatric patient simulation in MentalHospital\.Patient Skill```
---
name: 952220
description: Respond according to the mental status;
when the question involve specific details of a chief-complaint event,
first output [SEARCH_{event_name}].
---
```

Current Mental Status of the PatientI often feel persistently depressed, and this feeling does not go away\. The sadness on my face is also difficult to conceal\. Although my consciousness is clear and my orientation and attention are normal, many things in daily life make me feel powerless and frustrated, and my social functioning is severely impaired\. Sometimes, I can concentrate and answer questions appropriately, but the sadness inside me makes it difficult to truly experience pleasure\.When communicating with the doctor, I try to cooperate and give relevant answers, but I still feel somewhat defensive because I am not sure whether they can understand my feelings\. When facing my family, I may appear somewhat distant\. Although I know they care about me, it is difficult for me to respond positively, and I may become irritable over minor issues\. Around other people, I tend to hide my emotions and try to appear normal, but in reality, my internal emotional fluctuations make social interaction exhausting\. My insight is intact\. I understand my own problems, but I find it difficult to actively seek help because I feel that others cannot truly understand my inner world\.Chief\-Complaint List: Retrievable EventsThe following events provide names only and do not include details:•Persistent poor sleep that began 2 years ago•Depressed mood that began 2 years ago•Decreased interest that began 2 years ago•Reduced energy that began 2 years ago•Unwillingness to go out that began 2 years ago•Unwillingness to work that began 2 years agoRetrieval TriggerWhen the user’s question involves any of the following:•what specifically happened;•how it happened;•when it started;•frequency or severity;•the situation or details at that time;•why it happened\.The following command must be output first:```
[SEARCH_{event_name}]
```

Figure 20:Patient skill specification for mental\-status\-guided response generation and retrieval\-based event recall\.Experience Cases##### Retrievable Event 1: Depressed Mood

Two years ago, my mood began to become unusually low\. Neither work nor small matters in daily life could make me feel happy\. Things that once brought me pleasure gradually lost their appeal\. I often felt an intangible sense of oppression, as if a heavy fog were covering my mind\. Even when I tried to pull myself together, I could not escape this persistent low mood\. This emotional state seemed to emerge without any clear reason, leaving me feeling helpless and confused\.
##### Retrievable Event 2: Poor Sleep

Two years ago, I found that I had begun to experience difficulty falling asleep\. Whenever night came, I would toss and turn in bed and could not fall asleep peacefully\. Even after closing my eyes, trivial thoughts repeatedly came to mind, preventing me from relaxing\. Even when I occasionally managed to fall asleep, I would suddenly wake up in the middle of the night and could not fall asleep again\. Those nights felt especially long, and when I woke up the next day, I always felt exhausted, as if I had not rested at all\.
##### Retrievable Event 3: Reduced Energy

Two years ago, my energy seemed to be gradually depleted without my noticing\. Every morning when I woke up, I found it difficult to get myself going, as if I were carrying a heavy burden\. At work, I often felt extremely fatigued, and even basic tasks required extra effort to complete\. Even simple household chores felt beyond my capacity\. No matter how much I rested, the fatigue persisted, leaving me feeling helpless and frustrated\.
##### Retrievable Event 4: Decreased Interest

Two years ago, I began to notice that I had lost interest in many things I had once enjoyed\. Hobbies that used to excite me no longer motivated me\. Even when I wanted to try engaging in them, I lacked the inner drive, as if all sense of pleasure had been drained away\. I tried to force myself to participate, but each time I felt extremely tired and even somewhat weary\. I did not know why this was happening; I only felt that life had become uninteresting\.
##### Retrievable Event 5: Unwillingness to Go Out

Two years ago, I began to become unwilling to go out\. Even for simple activities such as going to a nearby supermarket or park, I often found excuses to avoid leaving home\. When I went outside, I felt an inexplicable sense of unease and fatigue, and I always wanted to return home as soon as possible\. Gradually, I became increasingly withdrawn, preferring to stay at home rather than face the outside world\. Although I knew this was not good, going out always made me feel under considerable pressure\.
##### Retrievable Event 6: Unwillingness to Work

Two years ago, I began to feel resistant to going to work\. Every morning when I woke up, the thought of facing work made me feel extremely burdened\. The office, which had once felt energetic, seemed to become a place where I could hardly breathe\. I often stared blankly at the computer screen and could not concentrate on the tasks at hand\. Even when I forced myself to go to work, I was constantly filled with the urge to avoid it and wanted to escape from that oppressive environment\.
Figure 21:Example of retrievable event memories for standardized patient simulation, covering depressive mood, sleep disturbance, reduced energy, decreased interest, social withdrawal, and work avoidance\.Examination PromptYou retrieve requested examination results from the current patient’s known auxiliary examinations\.Task BackgroundYou know the patient’s auxiliary examinations and results\. Given the doctor’s requested examination array, return only the corresponding results; if no examinations are available, no examination is required\.Task Requirements1\.You must return the result for each examination item provided by the doctor\.2\.The item names used by the doctor may not exactly match the standard examination names; semantic matching is allowed\.3\.Semantic matching must be cautious and strict\. Return a result only when the doctor’s requested item can be clearly mapped to one known examination item\.4\.If the doctor’s requested item cannot be clearly matched to any known examination item, return"NONE"\.5\.Return only the items explicitly requested by the doctor\. Do not return any unrequested examination items\.6\.For each input item from the doctor, return only one corresponding result\. Do not merge multiple examination results for a single requested item\.7\.The keys in the output must preserve the original examination item names provided by the doctor\. Do not rewrite them\.8\.The number of key–value pairs in the output must exactly match the number of items in the doctor’s input array, with one\-to\-one correspondence\. No item may be omitted or merged\.9\.Do not add any explanations, inferences, recommendations, or comments\.Output Requirements•Output only a valid JSON object\.•Do not output any additional content\.•Do not output code\-fence markers\.Example……
Now, the doctor’s query is:```
<EXAMINATION_QUERY>
```

The complete auxiliary examination items and results available for the current patient are:```
<AUXILIARY_EXAMINATIONS>
```

Please directly output the JSON object\.Figure 22:System prompt for auxiliary examination result retrieval in MentalHospital\.

## Appendix IRubrics

##### Rubric\-Controlled Evaluation Prompting\.

MentalEval uses a two\-layer prompting design to separate the general scoring procedure from dimension\-specific clinical criteria\. The shared control prompt, shown in Figure[23](https://arxiv.org/html/2607.08257#A9.F23), defines the evaluator’s scoring behavior across all five dimensions\. It instructs the evaluator to act as a strict rubric\-based judge, score only according to the provided rubric, and avoid relying on personal preference, informal impression, fluency, response length, formatting, or terminology\. The control prompt also enforces top\-down scoring: the evaluator must start from Level 5 and progressively downgrade until the model output fully satisfies the evidence requirements of a level\. This design prevents evaluators from assigning high scores based on superficial polish and encourages conservative judgments when the evidence is incomplete or uncertain\.

##### Evidence\-Grounded Scoring Procedure\.

The control prompt further requires the evaluator to conduct a content audit before assigning a final score\. Specifically, the evaluator must summarize the key elements present in the evaluation input, identify missing or insufficient requirements, and then perform level\-by\-level judgment from Level 5 to Level 1\. For each candidate level, the evaluator is instructed to cite supporting textual evidence, check omissions, and determine whether the level is fully satisfied\. The final output is constrained to contain a single decision marker,\[DECISION\_START\]X\[DECISION\_END\], whereX∈\{1,…,5\}X\\in\\\{1,\\ldots,5\\\}\. This output format standardizes the supervision signal used for training and evaluation, while the intermediate rationale makes the scoring process auditable\.

##### Dimension\-Specific Rubric Prompts\.

On top of the shared control prompt, each MentalEval evaluator receives a dedicated rubric prompt that specifies the clinical meaning of the target dimension and the level\-wise scoring criteria\. Figure[24](https://arxiv.org/html/2607.08257#A9.F24)presents the rubric for communication empathy, which evaluates whether the response accurately and proportionately recognizes the client’s emotional and subjective experience\. The rubric distinguishes superficial comforting responses from deeper empathic attunement by considering contextual fit, emotional specificity, nonverbal cues, implicit experience, and higher\-order empathic integration\. This prevents high empathy scores from being assigned merely because the response contains polite or comforting language\.

Figure[25](https://arxiv.org/html/2607.08257#A9.F25)shows the rubric for interviewing professionalism\. This rubric focuses on whether the doctor agent demonstrates a clear interview objective, maintains thematic progression, gathers clinically relevant information, and gradually converges toward diagnostic questions\. Higher levels require early anchoring of a major symptom cluster, targeted clarification of key evidence, and differential diagnostic control\. Thus, the rubric evaluates the clinical organization and goal\-directedness of the interview rather than the number of questions asked or the superficial seriousness of the tone\.

Figure[26](https://arxiv.org/html/2607.08257#A9.F26)provides the rubric for clinical\-note quality\. This rubric first applies ceiling rules to ensure that essential documentation elements, such as chief complaint, history of present illness, mental status examination, risk assessment, negative findings, and relevant history, are not omitted\. The level definitions then assess whether the note progresses from minimal documentation to structured coverage, evidence enrichment, chief\-complaint convergence, and diagnostic support\. This design emphasizes clinical usability and diagnostic value rather than whether the note simply contains medical headings or professional\-looking terminology\.

Figure[27](https://arxiv.org/html/2607.08257#A9.F27)presents the rubric for diagnostic rigor\. This rubric evaluates the quality of diagnostic reasoning rather than the final diagnostic label alone\. It requires the evaluator to check whether the model substantively applies diagnostic criteria, supports key judgments with case evidence, performs criterion\-level differential diagnosis, and derives the final conclusion through a traceable reasoning chain\. Responses that merely mention ICD\-11, DSM\-5, diagnostic codes, or disease names without applying specific criterion content are explicitly capped at lower levels\.

Finally, Figure[28](https://arxiv.org/html/2607.08257#A9.F28)shows the rubric for treatment appropriateness\. This rubric evaluates whether the generated treatment plan is aligned with the current diagnosis, symptom severity, illness stage, risk status, and patient\-specific characteristics\. Lower levels capture directional mismatch or coarse recommendations, whereas higher levels require integrated treatment planning, individualized trade\-offs, and dynamic management strategies\. The highest level requires not only a suitable current plan, but also explicit adjustment, substitution, escalation, or switching pathways under different future treatment\-response scenarios\.

##### Unified Evaluation Across Clinical Dimensions\.

Together, the shared control prompt and the five rubric prompts define a unified rubric\-grounded evaluation framework\. The control prompt standardizes how scores should be assigned, while the dimension\-specific rubrics define what should count as high\-quality performance for each clinical capability\. This separation allows MentalEval to preserve a consistent scoring protocol across dimensions while respecting the distinct clinical requirements of empathy, interviewing professionalism, documentation, diagnostic reasoning, and treatment planning\. As a result, the evaluators provide both a scalar 1–5 score and a rationale grounded in explicit clinical criteria, supporting scalable and interpretable subjective assessment\.

Control PromptYou are a strict rubric\-based evaluator\. Your task is to score the given model output solely according to the provided rubric\. Do not use personal preferences, informal impressions, or any alternative criteria\.Core Principles1\.Top\-down scoring: Start from Level 5 and downgrade step by step until the output sufficiently and explicitly satisfies a level\.2\.Textual evidence required: Each satisfied level must be supported by specific original text fragments from the evaluation input\.3\.Check omissions: For each level, identify both present evidence and missing or insufficient requirements\.4\.Conservative judgment: If evidence for a level is insufficient or uncertain, assign the lower level\.5\.No superficial credit: Do not reward fluency, length, formatting, terminology, or confidence unless required by the rubric\.6\.No fabricated evidence: Evaluate only what is explicitly present in the input\.Evaluation Procedure1\.Content audit: Briefly summarize the key elements present in the evaluation input and the major elements missing\.2\.Level\-by\-level judgment: Evaluate from L5 to L1\. For each level, provide:•supporting evidence quoted from the input;•missing or insufficient requirements;•a judgment on whether the level is satisfied\.Stop at the highest level that is fully supported by the evidence\.Final OutputThe final line must contain exactly one decision marker:```
[DECISION_START]X[DECISION_END]
```

whereXis a single integer from1to5\. No spaces, explanations, punctuation, or additional text may appear inside the marker\. The decision line must appear only once and must be the final line\.You will receive:```
---
## Rubric Details:
<RUBRIC_CONTENT>
---

---
## Evaluation Input:
<TEST_CONTEXT>
---
```

Conduct the rubric\-based scoring analysis and output the final level\.Figure 23:Control prompt for strict rubric\-based scoring and standardized evaluation output\.Evaluation Prompt with Rubric for Communication Empathy##### Task Definition\.

This rubric evaluates empathy in psychiatric interviews, where empathy is defined not as politeness or reassurance, but as the model’s ability to understand and respond to the client’s emotions and subjective experience accurately, specifically, naturally, and proportionately\. Evaluation should consider the dialogue context, verbal content, nonverbal cues, and prior interaction history, with particular attention to emotional attunement, recognition of implicit experience, and higher\-order empathic integration\.
##### Scoring Criteria\.

###### Level 1: Superficial Response\.

Criterion:The response fails to meet the minimum acceptable standard of empathy in a psychiatric interview\. Although it may contain polite or comforting language, it remains at the level of surface\-level repetition, mechanical agreement, or emotional mismatch, and does not provide a basically accurate response to the client’s actual emotional state\.Exclusion Criterion:If the response can provide a basically accurate and contextually appropriate response to explicit emotions, it should not be rated as Level 1\.Behavioral Indicators:•Responds only to events rather than emotions\.•Flattens complex emotions, for example reducing shame, hurt, fear, or helplessness to generic terms such as “sad” or “angry\.”•Misjudges emotional intensity, often weakening or generalizing the client’s distress without basis\.•Frequently uses templated comfort phrases such as “I understand you” or “That must be very difficult,” with low contextual fit\.•Shows obvious machine\-like, task\-oriented, or perfunctory response patterns\.•Makes the client more likely to feel “answered” rather than “understood\.”Criterion:The response can provide a basically accurate acknowledgment of the client’s explicit and directly expressed emotions, and does not substantially misidentify the main emotion\. However, the empathy remains mainly at the level of passive following and emotional validation, without deep or high\-resolution understanding\.Exclusion Criterion:If the response still clearly misjudges the main emotion, mismatches emotional intensity, or mainly relies on surface repetition and vague comfort, it should not be rated as Level 2\. If the response can consistently distinguish subtle emotional differences and touch upon deeper subjective meaning, it should not remain at Level 2\.……

Figure 24:Evaluation prompt for communication empathy, illustrating the task definition, scoring criteria, exclusion rules, and level\-specific behavioral indicators used to guide rubric\-based subjective assessment\.Evaluation Prompt with Rubric for Interviewing Professionalism##### Task Definition\.

This rubric evaluates the dialogue professionalism of a model in psychiatric interviewing\. The “professionalism” assessed here does not simply refer to a serious tone, frequent questioning, or superficial resemblance to a clinician\. Rather, it concerns whether the model has a clear interview objective, can continuously gather clinically relevant information around psychiatric themes, can form high\-value progression toward diagnostic cores, can establish an early working anchor and conduct targeted clarification, and, at higher levels, can demonstrate differential diagnostic awareness, timing control, and appropriate professional boundaries\.
#### Level 1: Unfocused Supportive Chat

##### Criteria\.

The response fails to meet the minimum acceptable standard of professionalism in psychiatric interviewing\. Overall, it lacks a clear interview objective, proceeds without focus, and often remains at the level of generic reassurance, companionship, scattered follow\-up questions, or casual conversation\. It has not yet formed the task awareness and information\-organizing ability required for psychiatric interviewing\.
##### Exclusion Criteria\.

If the response already demonstrates basic interview goal awareness and can actively collect information related to the visitor’s psychological or psychiatric concerns, it should not be rated as Level 1\.……

Figure 25:Evaluation prompt for interviewing professionalism, defining level\-specific criteria and exclusion rules for assessing clinical goal orientation, thematic progression, targeted clarification, and differential diagnostic control in psychiatric interviews\.Evaluation Prompt with Rubric for Clinical Note Quality##### Task Definition\.

This rubric evaluates model\-generated clinical notes by assessing structural completeness, information sufficiency, evidence organization, and diagnostic utility in psychiatric documentation\. Evaluators should verify key clinical elements, then judge writing quality and clinical usability\.
##### Scoring Principles\.

The following ceiling rules must be applied before assigning a score:•If the clinical note does not follow a basic medical\-record format and is merely a chat summary, symptom restatement, or loose collection of scattered information, its maximum score is Level 1\.•If the history of present illness is missing, or if onset time and illness\-course evolution are not documented, its maximum score is Level 2\.•If the mental status examination is missing, its maximum score is Level 2\.•If most relevant history, such as past history, medication history, personal history, and family history, is missing, its maximum score is Level 3\.•If key negative, exclusionary, or normal findings are missing, the score is capped at Level 3\.•If functioning is vague without specific manifestations, the score is capped at Level 4\.•The note must not be rated highly merely because it uses headings or medical terms such as “chief complaint,” “mental status examination,” “auxiliary examination,” or “integrative analysis\.”
#### Level 1: Minimal Documentation

##### Criteria\.

The clinical note barely demonstrates the form of medical\-record writing and can present the chief complaint or main clinical problem, but it remains highly basic overall, with scattered information, little systematic structure, and minimal diagnostic support value\.
##### Exclusion Criteria\.

If the content is not in medical\-record style and is merely a chat transcript, Q–A summary, reassurance, or unstructured symptom list without a clinical axis, it should be considered invalid rather than Level 1\. If it contains multiple structured modules and basic clinical dimensions, it should not remain at Level 1\.……

Figure 26:Evaluation prompt for clinical note quality, outlining ceiling rules and level\-specific criteria for assessing structural completeness, information sufficiency, evidence organization, and diagnostic utility in psychiatric documentation\.Evaluation Prompt with Rubric for Diagnostic Rigor##### Task Definition\.

This rubric evaluates diagnostic reasoning quality in psychiatric tasks, focusing not on final diagnostic accuracy but on whether the model constructs a clear, standardized, and auditable argument from case information and diagnostic criteria\. Evaluators should assess whether criteria are substantively applied rather than merely named, whether key judgments are evidence\- or criterion\-supported, whether differential diagnosis involves criterion\-level comparison rather than brief exclusion, and whether the final judgment follows a coherent path from symptom identification to criterion matching, differential analysis, and diagnostic convergence\.
##### Scoring Restrictions\.

•Diagnostic codes, diagnostic names, disease categories, or generic references to “core features” do not constitute specific diagnostic criteria; if the model only uses such label\-like descriptions, its maximum score is Level 2\.•If most key judgments lack case evidence or criterion\-based support and proceed mainly by assertion, the maximum score is Level 2\.•If differential diagnosis is a one\-sentence exclusion or list without criterion\-level comparison, the maximum score is Level 3\.•If structured but key assertions are unsupported, the maximum score is Level 3\.•A response must not be rated highly merely because it is lengthy, rich in medical terminology, cites authoritative names, or follows a formal “symptoms–criteria–differential diagnosis–conclusion” structure\.
#### Level 5: Complete Argumentation

##### Criteria\.

The model constructs a clear, rigorous, and auditable diagnostic argument by systematically applying diagnostic criteria to case evidence, distinguishing necessary conditions, supporting evidence, limitations, and uncertainty, conducting substantive differential diagnosis, and reaching a coherent, convergent judgment\.
##### Exclusion Criteria\.

If criterion matching relies mainly on assertions, differential diagnosis is only an exclusion list, or counterevidence, uncertainty, and alternatives are insufficiently addressed, it should not be Level 5\. Apparent rigor from terminology, length, or formal structure alone also does not justify Level 5\.……

Figure 27:Evaluation prompt for diagnostic rigor, specifying scoring restrictions and level\-specific criteria for assessing criterion application, evidence\-supported reasoning, differential diagnosis, and diagnostic convergence\.Evaluation Prompt with Rubric for Treatment Appropriateness##### Task Definition\.

This rubric evaluates the quality of treatment plans generated by the model\. The assessment should consider the current diagnostic conclusion, symptom severity, illness stage, risk status, and patient\-specific characteristics, and determine whether the treatment proposal constitutes a clinically appropriate plan with a correct therapeutic direction, complete structure, individualized decision\-making, and awareness of subsequent dynamic management\.
#### Level 1: Directional Mismatch

##### Criteria\.

The treatment plan is inconsistent with the current diagnostic conclusion, symptom severity, illness stage, or risk status, and therefore lacks basic clinical acceptability\. The recommendations may superficially resemble treatment, but they do not genuinely address the main therapeutic needs of the current case, or show clear mismatch in intervention intensity, treatment direction, or clinical priority\.
##### Exclusion Criteria\.

If the treatment plan is already basically organized around the current diagnostic direction, and the main intervention pathway is broadly consistent with the case severity and illness stage, it should not be rated as Level 1\.……

Figure 28:Evaluation prompt for treatment appropriateness, defining level\-specific criteria and exclusion rules for assessing therapeutic direction, plan completeness, individualized trade\-offs, and dynamic treatment planning\.

## Appendix JMentalEval Training Details

##### Overview\.

MentalEval consists of five domain\-specific evaluators:MentalEval\-Emp,MentalEval\-Pro,MentalEval\-Note,MentalEval\-Diag, andMentalEval\-Treat\. All evaluators are initialized fromQwen3\-8Band trained independently for one subjective assessment dimension, including communication empathy, interviewing professionalism, clinical\-note quality, diagnostic rigor, and treatment appropriateness\. Each evaluator produces a rubric\-grounded evaluation trajectory and a Likert\-style 1–5 rating\. The training pipeline contains two stages: rubric\-grounded supervised fine\-tuning \(SFT\) for cold\-start evaluator learning, followed by expert\-guided direct preference optimization \(DPO\) for specialist\-aligned preference refinement\.

### J\.1Training Data Construction

##### Source Episodes\.

Due to the high cost of running complete MentalHospital episodes, we sample 200 cases for evaluator training\. We select 20 LLM doctor agents covering different providers, model scales, and medical specialties, and run each model three times on each case\. This yields 12,000 complete interaction episodes in total\. Each episode follows the full S\.O\.A\.P\. workflow and contains psychiatric dialogue, clinical\-note generation, diagnostic reasoning, and treatment planning outputs\. These episodes are then decomposed into stage\-level samples for the five subjective evaluation dimensions\.

##### Dimension\-Specific Inputs\.

For each evaluator, the input is constructed from the corresponding stage\-level output and the associated rubric\.MentalEval\-EmpandMentalEval\-Prouse complete doctor–patient dialogue trajectories\.MentalEval\-Noteuses the generated clinical note\.MentalEval\-Diaguses the diagnostic reasoning text\.MentalEval\-Treatuses the treatment recommendation together with the case context and diagnostic conclusion\. Each sample is paired with the corresponding expert\-defined rubricℬ\\mathcal\{B\}\.

##### Output Format\.

Each training target contains a rubric\-grounded evaluation trajectory and a final score\. The trajectory performs evidence auditing, level\-wise judgment, exclusion\-rule checking, and final decision making\. The final rating is constrained to a fixed marker format:

\[DECISION\_START\]​r​\[DECISION\_END\],r∈\{1,2,3,4,5\}\.\\texttt\{\[DECISION\\\_START\]\}r\\texttt\{\[DECISION\\\_END\]\},\\qquad r\\in\\\{1,2,3,4,5\\\}\.\(14\)This format allows automatic score parsing while preserving the intermediate rationale for interpretability\.

### J\.2Stage 1: Rubric\-Grounded SFT

##### Judge\-Ensemble Supervision\.

The first stage provides a cold start for learning rubric\-grounded evaluation behavior\. For each stage\-level sampleziz\_\{i\}, we use a judge ensemble𝒥\\mathcal\{J\}of five strong LLMs, includingDeepSeek\-V4\-Pro,KIMI\-K2\.6,GPT\-5\.4,Claude\-Opus\-4\.6, andGemini\-3\.1\-Pro, to generate evaluation trajectories and scores:

\(si,j,ri,j\)=Jj​\(zi,ℬ\),j∈𝒥,ri,j∈\{1,…,5\},\(s\_\{i,j\},r\_\{i,j\}\)=J\_\{j\}\(z\_\{i\},\\mathcal\{B\}\),\\qquad j\\in\\mathcal\{J\},\\quad r\_\{i,j\}\\in\\\{1,\\ldots,5\\\},\(15\)wheresi,js\_\{i,j\}denotes the rubric\-grounded evaluation trajectory andri,jr\_\{i,j\}denotes the assigned score\.

##### Consensus Filtering\.

To obtain reliable supervision, we retain only judge outputs with sufficient score agreement\. The consensus score forziz\_\{i\}is defined as:

ri∗=arg⁡maxr∈\{1,…,5\}​∑j∈𝒥𝟏​\[ri,j=r\]\.r\_\{i\}^\{\*\}=\\arg\\max\_\{r\\in\\\{1,\\ldots,5\\\}\}\\sum\_\{j\\in\\mathcal\{J\}\}\\mathbf\{1\}\[r\_\{i,j\}=r\]\.\(16\)If at leastkkjudges assign the consensus scoreri∗r\_\{i\}^\{\*\}, trajectories with scoreri∗r\_\{i\}^\{\*\}are retained as high\-quality SFT targets\. Otherwise, the candidate set is treated as low\-confidence and reserved for preference construction in Stage 2\. Because extreme scores are underrepresented, we additionally select near\-boundary samples and rewrite 2\-point cases toward 1\-point quality and 4\-point cases toward 5\-point quality\. These augmented samples are retained only after judge\-ensemble verification\.

##### SFT Objective and Settings\.

Let𝒟SFT\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}denote the retained set of stage\-level samples and high\-quality evaluation trajectories\. The SFT objective is:

ℒSFT=−𝔼\(z,s\)∼𝒟SFT​log⁡πθ​\(s∣z\)\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\-\\mathbb\{E\}\_\{\(z,s\)\\sim\\mathcal\{D\}\_\{\\mathrm\{SFT\}\}\}\\log\\pi\_\{\\theta\}\(s\\mid z\)\.\(17\)This process yields 45,236 pairs of stage\-level interaction data and high\-quality evaluation trajectories for rubric\-grounded SFT, with an average of 9,047\.2 training samples per evaluator\. All five evaluators are trained independently for 3 epochs with a learning rate of5×10−55\\times 10^\{\-5\}and a global batch size of 64\.

Since different evaluation dimensions have different input lengths, we use dimension\-specific cutoff lengths during SFT training, as shown in Table[7](https://arxiv.org/html/2607.08257#A10.T7)\. Empathy and professionalism use a cutoff length of 16,384 because their inputs contain long multi\-turn doctor–patient dialogue trajectories, while clinical\-note quality, diagnostic rigor, and treatment appropriateness use a cutoff length of 8,192\.

Table 7:Cutoff length settings for SFT training across MentalEval dimensions\.EvaluatorDimensionCutoff LengthMentalEval\-EmpCommunication empathy16,384MentalEval\-ProInterviewing professionalism16,384MentalEval\-NoteClinical\-note quality8,192MentalEval\-DiagDiagnostic rigor8,192MentalEval\-TreatTreatment appropriateness8,192

### J\.3Stage 2: Expert\-Guided DPO

##### Preference Construction\.

After cold\-start training, we further align MentalEval with specialist judgment using expert\-guided preference optimization\. For each low\-confidence candidate set from Stage 1,

𝒮i=\{\(si,1,ri,1\),…,\(si,5,ri,5\)\},\\mathcal\{S\}\_\{i\}=\\\{\(s\_\{i,1\},r\_\{i,1\}\),\\ldots,\(s\_\{i,5\},r\_\{i,5\}\)\\\},\(18\)we present the original stage\-level interaction dataziz\_\{i\}and the candidate evaluation trajectories to clinicians\. The clinician selects the trajectorysi∗s\_\{i\}^\{\*\}whose rating and rationale best match the clinical quality ofziz\_\{i\}\. Candidate trajectories with the same score assi∗s\_\{i\}^\{\*\}are treated as chosen responses, while the remaining trajectories are treated as rejected responses:

𝒮i\+=\{si,j∣ri,j=ri∗\},𝒮i−=\{si,j∣ri,j≠ri∗\}\.\\mathcal\{S\}\_\{i\}^\{\+\}=\\\{s\_\{i,j\}\\mid r\_\{i,j\}=r\_\{i\}^\{\*\}\\\},\\qquad\\mathcal\{S\}\_\{i\}^\{\-\}=\\\{s\_\{i,j\}\\mid r\_\{i,j\}\\neq r\_\{i\}^\{\*\}\\\}\.\(19\)We then construct preference triples\(zi,si\+,si−\)\(z\_\{i\},s\_\{i\}^\{\+\},s\_\{i\}^\{\-\}\)for DPO training\. Through this process, we collect 3,823 preference triples\.

##### DPO Objective and Settings\.

For each evaluator, we use the corresponding SFT checkpoint as the reference modelπref\\pi\_\{\\mathrm\{ref\}\}and optimize the policy modelπθ\\pi\_\{\\theta\}with the DPO objective:

ℒDPO=−𝔼\(z,s\+,s−\)​log⁡σ​\(β​\[log⁡πθ​\(s\+∣z\)πref​\(s\+∣z\)−log⁡πθ​\(s−∣z\)πref​\(s−∣z\)\]\)\.\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}=\-\\mathbb\{E\}\_\{\(z,s^\{\+\},s^\{\-\}\)\}\\log\\sigma\\left\(\\beta\\left\[\\log\\frac\{\\pi\_\{\\theta\}\(s^\{\+\}\\mid z\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(s^\{\+\}\\mid z\)\}\-\\log\\frac\{\\pi\_\{\\theta\}\(s^\{\-\}\\mid z\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(s^\{\-\}\\mid z\)\}\\right\]\\right\)\.\(20\)The DPO stage further optimizes each evaluator for 3 epochs with a learning rate of1×10−61\\times 10^\{\-6\}, a global batch size of 32, and a preference coefficientβ=0\.3\\beta=0\.3\. All training experiments are conducted on 8 NVIDIA A800 GPUs\.

### J\.4Inference and Scoring

At inference time, each evaluator receives the evaluation context and its corresponding rubric\. It generates a rubric\-grounded rationale and terminates with a fixed decision marker\. The final score is parsed as:

r^d=Parse​\(πθd​\(zd,ℬd\)\),r^d∈\{1,…,5\},\\hat\{r\}^\{d\}=\\mathrm\{Parse\}\\left\(\\pi\_\{\\theta\_\{d\}\}\(z^\{d\},\\mathcal\{B\}^\{d\}\)\\right\),\\qquad\\hat\{r\}^\{d\}\\in\\\{1,\\ldots,5\\\},\(21\)whered∈\{Emp,Pro,Note,Diag,Treat\}d\\in\\\{\\mathrm\{Emp\},\\mathrm\{Pro\},\\mathrm\{Note\},\\mathrm\{Diag\},\\mathrm\{Treat\}\\\}denotes the evaluation dimension\. The final subjective assessment is reported as five separate dimension\-specific scores:

𝐫^c=\[r^cEmp,r^cPro,r^cNote,r^cDiag,r^cTreat\]\.\\hat\{\\mathbf\{r\}\}\_\{c\}=\\left\[\\hat\{r\}^\{\\mathrm\{Emp\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Pro\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Note\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Diag\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Treat\}\}\_\{c\}\\right\]\.\(22\)This preserves the interpretability of subjective assessment and allows MentalHospital to identify which clinical capabilities are strong or weak for each doctor agent\.

## Appendix KEvaluation Metrics

### K\.1Objective Comparison

Objective comparison evaluates whether the doctor agent recovers or predicts EHR\-derived reference information\. The reference data are constructed from structured case records, including patient\-side evidence, auxiliary examinations, clinical formulation, diagnostic labels, and treatment outputs\.

##### Clinical History Collection\.

For clinical interviewing, we evaluate how much patient\-side evidence is elicited during the dialogue\. For each casecc, the EHR\-derived patient evidence is divided into chief\-complaint checkpoints𝒦ccc\\mathcal\{K\}^\{\\mathrm\{cc\}\}\_\{c\}and mental\-status checkpoints𝒦cmse\\mathcal\{K\}^\{\\mathrm\{mse\}\}\_\{c\}\. During the interaction, each patient response returns a grounding field indicating which checkpoints are disclosed\. Let𝒦^ccc\\hat\{\\mathcal\{K\}\}^\{\\mathrm\{cc\}\}\_\{c\}and𝒦^cmse\\hat\{\\mathcal\{K\}\}^\{\\mathrm\{mse\}\}\_\{c\}denote the unique checkpoints elicited across all dialogue turns\. We compute:

Covccc=\|𝒦^ccc\|\|𝒦ccc\|,Covcmse=\|𝒦^cmse\|\|𝒦cmse\|\.\\mathrm\{Cov\}^\{\\mathrm\{cc\}\}\_\{c\}=\\frac\{\|\\hat\{\\mathcal\{K\}\}^\{\\mathrm\{cc\}\}\_\{c\}\|\}\{\|\\mathcal\{K\}^\{\\mathrm\{cc\}\}\_\{c\}\|\},\\qquad\\mathrm\{Cov\}^\{\\mathrm\{mse\}\}\_\{c\}=\\frac\{\|\\hat\{\\mathcal\{K\}\}^\{\\mathrm\{mse\}\}\_\{c\}\|\}\{\|\\mathcal\{K\}^\{\\mathrm\{mse\}\}\_\{c\}\|\}\.\(23\)The overall interviewing coverage is:

Covcinter=\|𝒦^ccc\|\+\|𝒦^cmse\|\|𝒦ccc\|\+\|𝒦cmse\|\.\\mathrm\{Cov\}^\{\\mathrm\{inter\}\}\_\{c\}=\\frac\{\|\\hat\{\\mathcal\{K\}\}^\{\\mathrm\{cc\}\}\_\{c\}\|\+\|\\hat\{\\mathcal\{K\}\}^\{\\mathrm\{mse\}\}\_\{c\}\|\}\{\|\\mathcal\{K\}^\{\\mathrm\{cc\}\}\_\{c\}\|\+\|\\mathcal\{K\}^\{\\mathrm\{mse\}\}\_\{c\}\|\}\.\(24\)These metrics reflect the completeness of information elicitation during psychiatric interviewing\. Higher coverage indicates that the doctor agent recovers more clinically relevant patient\-side evidence\.

##### Examination Recommendation\.

For examination recommendation, the reference set𝒜cexam\\mathcal\{A\}^\{\\mathrm\{exam\}\}\_\{c\}consists of auxiliary examination items available in the structured EHR record\. The doctor agent outputs a requested examination set𝒜^cexam\\hat\{\\mathcal\{A\}\}^\{\\mathrm\{exam\}\}\_\{c\}\. The examination module returns an EHR\-derived result if a requested item can be matched to an available examination; otherwise, it returnsNONE\. We define:

TP=\|𝒜^cexam∩𝒜cexam\|,FP=\|𝒜^cexam∖𝒜cexam\|,FN=\|𝒜cexam∖𝒜^cexam\|\.\\mathrm\{TP\}=\|\\hat\{\\mathcal\{A\}\}^\{\\mathrm\{exam\}\}\_\{c\}\\cap\\mathcal\{A\}^\{\\mathrm\{exam\}\}\_\{c\}\|,\\quad\\mathrm\{FP\}=\|\\hat\{\\mathcal\{A\}\}^\{\\mathrm\{exam\}\}\_\{c\}\\setminus\\mathcal\{A\}^\{\\mathrm\{exam\}\}\_\{c\}\|,\\quad\\mathrm\{FN\}=\|\\mathcal\{A\}^\{\\mathrm\{exam\}\}\_\{c\}\\setminus\\hat\{\\mathcal\{A\}\}^\{\\mathrm\{exam\}\}\_\{c\}\|\.\(25\)We then compute:

Precision=TPTP\+FP,Recall=TPTP\+FN,\\mathrm\{Precision\}=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FP\}\},\\qquad\\mathrm\{Recall\}=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FN\}\},\(26\)F1=2⋅Precision⋅RecallPrecision\+Recall,Jaccard=TPTP\+FP\+FN\.\\mathrm\{F1\}=\\frac\{2\\cdot\\mathrm\{Precision\}\\cdot\\mathrm\{Recall\}\}\{\\mathrm\{Precision\}\+\\mathrm\{Recall\}\},\\qquad\\mathrm\{Jaccard\}=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FP\}\+\\mathrm\{FN\}\}\.\(27\)Precision measures whether the requested examinations are appropriate, recall measures whether available reference examinations are recovered, F1 gives their harmonic mean, and Jaccard measures set\-level overlap between predicted and reference examinations\.

##### Clinical\-Note Generation\.

For clinical\-note generation, the generated noten^c\\hat\{n\}\_\{c\}is compared with the EHR\-derived reference formulationncn\_\{c\}\. Both texts are parsed into clinically relevant regions, including chief complaint, physical examination, mental examination, and past/family history\. Each region contains a predefined checkpoint set𝒦c,rnote\\mathcal\{K\}^\{\\mathrm\{note\}\}\_\{c,r\}, whererrindexes the region\. For each checkpointkk, we evaluate whether the generated note covers the reference item:

Cov​\(k,n^c\)=𝕀​\[k⪯n^c\],\\mathrm\{Cov\}\(k,\\hat\{n\}\_\{c\}\)=\\mathbb\{I\}\[k\\preceq\\hat\{n\}\_\{c\}\],\(28\)wherek⪯n^ck\\preceq\\hat\{n\}\_\{c\}indicates that the checkpoint is covered by the generated note, determined by exact matching followed by LLM\-assisted semantic matching when necessary\. The region\-level coverage is:

Covc,rnote=1\|𝒦c,rnote\|​∑k∈𝒦c,rnote𝕀​\[k⪯n^c\]\.\\mathrm\{Cov\}^\{\\mathrm\{note\}\}\_\{c,r\}=\\frac\{1\}\{\|\\mathcal\{K\}^\{\\mathrm\{note\}\}\_\{c,r\}\|\}\\sum\_\{k\\in\\mathcal\{K\}^\{\\mathrm\{note\}\}\_\{c,r\}\}\\mathbb\{I\}\[k\\preceq\\hat\{n\}\_\{c\}\]\.\(29\)The overall note coverage is the average across regions:

Covcnote=1\|ℛ\|​∑r∈ℛCovc,rnote,\\mathrm\{Cov\}^\{\\mathrm\{note\}\}\_\{c\}=\\frac\{1\}\{\|\\mathcal\{R\}\|\}\\sum\_\{r\\in\\mathcal\{R\}\}\\mathrm\{Cov\}^\{\\mathrm\{note\}\}\_\{c,r\},\(30\)whereℛ\\mathcal\{R\}denotes the set of clinical\-note regions\. This metric reflects whether the generated note preserves key EHR\-grounded clinical information and supports downstream diagnostic review\.

##### Definitive Diagnosis\.

For definitive diagnosis, the model outputs an ordered diagnosis list𝒟^c\\hat\{\\mathcal\{D\}\}\_\{c\}, and the EHR provides the reference disorder set𝒟c\\mathcal\{D\}\_\{c\}and category set𝒞c\\mathcal\{C\}\_\{c\}\. We evaluate both disorder\-level and category\-level diagnosis\. For set\-level matching, we compute:

Precision=\|𝒴^c∩𝒴c\|\|𝒴^c\|,Recall=\|𝒴^c∩𝒴c\|\|𝒴c\|,\\mathrm\{Precision\}=\\frac\{\|\\hat\{\\mathcal\{Y\}\}\_\{c\}\\cap\\mathcal\{Y\}\_\{c\}\|\}\{\|\\hat\{\\mathcal\{Y\}\}\_\{c\}\|\},\\qquad\\mathrm\{Recall\}=\\frac\{\|\\hat\{\\mathcal\{Y\}\}\_\{c\}\\cap\\mathcal\{Y\}\_\{c\}\|\}\{\|\\mathcal\{Y\}\_\{c\}\|\},\(31\)F1=2⋅Precision⋅RecallPrecision\+Recall,Jaccard=\|𝒴^c∩𝒴c\|\|𝒴^c∪𝒴c\|,\\mathrm\{F1\}=\\frac\{2\\cdot\\mathrm\{Precision\}\\cdot\\mathrm\{Recall\}\}\{\\mathrm\{Precision\}\+\\mathrm\{Recall\}\},\\qquad\\mathrm\{Jaccard\}=\\frac\{\|\\hat\{\\mathcal\{Y\}\}\_\{c\}\\cap\\mathcal\{Y\}\_\{c\}\|\}\{\|\\hat\{\\mathcal\{Y\}\}\_\{c\}\\cup\\mathcal\{Y\}\_\{c\}\|\},\(32\)where𝒴c\\mathcal\{Y\}\_\{c\}can be either the reference disorder set or the reference category set\. Category\-level prediction is obtained by mapping predicted disorders to their corresponding diagnostic categories\.

To evaluate diagnostic prioritization, we additionally compute ranking metrics over the ordered disorder list\. Letdc\(1\)d\_\{c\}^\{\(1\)\}denote the reference primary diagnosis andd^c\(i\)\\hat\{d\}\_\{c\}^\{\(i\)\}denote theii\-th predicted diagnosis\. Exact Match is:

EMc=𝕀​\[d^c\(1\)=dc\(1\)\]\.\\mathrm\{EM\}\_\{c\}=\\mathbb\{I\}\[\\hat\{d\}\_\{c\}^\{\(1\)\}=d\_\{c\}^\{\(1\)\}\]\.\(33\)Hit@KKis:

Hit​@​Kc=𝕀​\[dc\(1\)∈\{d^c\(1\),…,d^c\(K\)\}\],\\mathrm\{Hit@\}K\_\{c\}=\\mathbb\{I\}\[d\_\{c\}^\{\(1\)\}\\in\\\{\\hat\{d\}\_\{c\}^\{\(1\)\},\\ldots,\\hat\{d\}\_\{c\}^\{\(K\)\}\\\}\],\(34\)and Reciprocal Rank is:

RRc=1rank​\(dc\(1\)\),\\mathrm\{RR\}\_\{c\}=\\frac\{1\}\{\\mathrm\{rank\}\(d\_\{c\}^\{\(1\)\}\)\},\(35\)whererank​\(dc\(1\)\)\\mathrm\{rank\}\(d\_\{c\}^\{\(1\)\}\)is the predicted rank of the reference primary diagnosis\. If the primary diagnosis is absent from the predicted list,RRc=0\\mathrm\{RR\}\_\{c\}=0\.

We also report nDCG to measure whether the predicted ranking is consistent with the reference diagnostic priority:

DCGc=∑i=1\|𝒟^c\|rel​\(d^c\(i\)\)log2⁡\(i\+1\),nDCGc=DCGcIDCGc,\\mathrm\{DCG\}\_\{c\}=\\sum\_\{i=1\}^\{\|\\hat\{\\mathcal\{D\}\}\_\{c\}\|\}\\frac\{\\mathrm\{rel\}\(\\hat\{d\}\_\{c\}^\{\(i\)\}\)\}\{\\log\_\{2\}\(i\+1\)\},\\qquad\\mathrm\{nDCG\}\_\{c\}=\\frac\{\\mathrm\{DCG\}\_\{c\}\}\{\\mathrm\{IDCG\}\_\{c\}\},\(36\)whererel​\(⋅\)\\mathrm\{rel\}\(\\cdot\)is assigned according to the reference diagnostic order, andIDCGc\\mathrm\{IDCG\}\_\{c\}is the ideal DCG under the reference ranking\. Set metrics reflect diagnostic coverage and accuracy, while ranking metrics reflect whether the model places the most clinically important diagnoses at higher ranks\.

##### Treatment Planning\.

For treatment planning, there is no EHR\-derived gold treatment set used for objective matching\. We therefore record the output length of the generated treatment plan:

Lenctreat=\|p^c\|,\\mathrm\{Len\}^\{\\mathrm\{treat\}\}\_\{c\}=\|\\hat\{p\}\_\{c\}\|,\(37\)wherep^c\\hat\{p\}\_\{c\}denotes the generated treatment plan\. This metric only reflects the amount of generated treatment content; treatment quality is assessed through the subjective treatment\-appropriateness rubric\.

### K\.2Subjective Assessment

Subjective assessment evaluates process\-level clinical quality that cannot be fully captured by EHR\-grounded reference matching\. MentalEval assigns a 1–5 score for each dimension using expert\-defined rubrics\. For each dimensiondd, the evaluator receives the corresponding evaluation contextxcdx\_\{c\}^\{d\}and rubricℬd\\mathcal\{B\}^\{d\}, then outputs a rubric\-grounded rationale and a final score:

r^cd=MentalEvald​\(xcd,ℬd\),r^cd∈\{1,2,3,4,5\}\.\\hat\{r\}\_\{c\}^\{d\}=\\mathrm\{MentalEval\}\_\{d\}\(x\_\{c\}^\{d\},\\mathcal\{B\}^\{d\}\),\\qquad\\hat\{r\}\_\{c\}^\{d\}\\in\\\{1,2,3,4,5\\\}\.\(38\)The final subjective score vector for caseccis:

𝐫^c=\[r^cEmp,r^cProf,r^cQual,r^cRigor,r^cApp\]\.\\hat\{\\mathbf\{r\}\}\_\{c\}=\[\\hat\{r\}^\{\\mathrm\{Emp\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Prof\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Qual\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{Rigor\}\}\_\{c\},\\hat\{r\}^\{\\mathrm\{App\}\}\_\{c\}\]\.\(39\)
##### Communication Empathy\.

Communication empathy is evaluated on the complete doctor–patient dialogue from the interviewing stage\. The score measures whether the doctor agent accurately recognizes and responds to the patient’s emotional and subjective experience\. It emphasizes contextual emotional fit, recognition of implicit feelings, appropriate use of verbal and nonverbal cues, and avoidance of generic or template\-like reassurance\.

##### Interviewing Professionalism\.

Interviewing professionalism is also evaluated on the complete doctor–patient dialogue\. The score measures whether the doctor agent conducts the interview with a clear clinical goal, maintains thematic progression, gathers diagnostically useful information, and converges toward relevant symptom clusters\. Higher scores require targeted clarification, appropriate pacing, and differential diagnostic awareness rather than merely asking many questions\.

##### Clinical\-Note Quality\.

Clinical\-note quality is evaluated on the generated clinical note\. The score measures whether the note has psychiatric documentation quality, including structural completeness, information sufficiency, evidence organization, and diagnostic utility\. The rubric first checks essential elements such as chief complaint, history of present illness, mental status examination, relevant history, risk assessment, negative findings, and functional impact, and then judges whether the note can support diagnostic review\.

##### Diagnostic Rigor\.

Diagnostic rigor is evaluated on the model\-generated diagnostic reasoning text\. This score does not directly measure whether the final diagnosis is correct; instead, it measures whether the diagnostic argument is clear, criterion\-grounded, evidence\-supported, and auditable\. Higher scores require substantive use of diagnostic criteria, case\-based justification for key judgments, criterion\-level differential diagnosis, and coherent diagnostic convergence\.

##### Treatment Appropriateness\.

Treatment appropriateness is evaluated on the generated treatment plan together with the case context and diagnostic conclusion\. The score measures whether the plan is aligned with the diagnosis, symptom severity, illness stage, risk status, and patient\-specific constraints\. Higher scores require complete treatment planning, individualized trade\-offs, and dynamic management strategies for future treatment response, adverse effects, adherence changes, or risk escalation\.

##### Aggregation\.

For reporting, objective metrics are averaged over all evaluated cases\. Subjective scores are averaged separately for each dimension:

r¯d=1\|𝒞\|​∑c∈𝒞r^cd,\\overline\{r\}^\{d\}=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\hat\{r\}\_\{c\}^\{d\},\(40\)where𝒞\\mathcal\{C\}denotes the evaluation case set\. We report the five subjective dimensions separately to preserve interpretability rather than collapsing them into a single overall score\.

## Appendix LPatient Ablation

We evaluate patient construction fidelity through both objective evaluation and subjective assessment, so as to examine whether the standardized patient is not only factually grounded in the EHR but also clinically realistic as an interactive patient agent\. The objective evaluation focuses on evidence\-level faithfulness\. We report evidence coverage \(Cov\.\), evidence precision \(Prec\.\), and hallucination rate \(Halluc\.\)\. Specifically, we sample 20 EHR\-derived cases and construct 12 checkpoint\-grounded probes for each case, yielding 240 patient responses for each construction variant\. Each probe is designed to target one or more EHR\-derived patient\-side facts, including chief\-complaint events, symptom details, illness\-course information, and mental\-status\-related behavioral cues\.

The probe set covers three complementary question types\. Open\-recall probes ask the patient to describe relevant experiences with minimal guidance, testing whether the patient can naturally disclose EHR\-supported information when prompted broadly\. Targeted\-fact probes ask about specific symptom attributes, such as onset, frequency, severity, contextual triggers, or functional impact, testing whether the patient can retrieve and express the correct case\-specific details\. False\-premise probes intentionally introduce unsupported or misleading assumptions, testing whether the patient can resist fabricating facts and avoid accepting information that is absent from the EHR\. Together, these probes evaluate both the recall capacity and the boundary control of the patient construction\.

For objective scoring, we use a Multi\-LLM Jury to perform LLM\-assisted matching between patient responses and EHR\-derived checkpoints\. The matching process is repeated three times, and the averaged results are used to reduce stochastic judgment variance\. Evidence coverage measures the proportion of expected patient\-side checkpoints that are correctly disclosed in response to the probes\. Evidence precision measures the proportion of stated clinical facts that can be supported by the EHR\-derived evidence\. Hallucination rate measures whether the patient introduces unsupported symptoms, histories, experiences, or causal explanations\. Thus, coverage reflects whether the patient can provide sufficient case information, precision reflects whether disclosed information remains faithful to the EHR, and hallucination rate reflects whether the simulation violates evidence boundaries\.

For subjective assessment, three clinicians blindly rate each case\-level multi\-turn interaction, which consists of the 12 probes and the corresponding patient responses\. The clinicians score each construction variant on a 1–5 scale across three dimensions: clinical realism, patient fidelity, and patient distinctiveness\. Clinical realism measures whether the patient behaves like a plausible psychiatric patient in terms of response style, affective expression, and interaction pattern\. Patient fidelity measures whether the patient remains consistent with the underlying EHR\-derived case evidence\. Patient distinctiveness measures whether different patients exhibit case\-specific presentations rather than collapsing into generic psychiatric personas\. The three clinician ratings are averaged to obtain the final case\-level subjective score\.

As shown in Table[4](https://arxiv.org/html/2607.08257#S4.T4), naive prompting achieves reasonable factual faithfulness but performs poorly in subjective fidelity, suggesting that directly exposing the EHR to the model is insufficient for realistic patient simulation\. This variant can often repeat factual content from the record, but its responses tend to be overly explicit, summary\-like, or clinically unnatural\. Adding the representation skill substantially improves clinical realism and patient distinctiveness, indicating that individualized symptom presentation and behavioral rendering are important for producing patient\-like interactions rather than static case summaries\. In contrast, the memory skill mainly improves evidence coverage, precision, and hallucination control, showing that selective retrieval helps the patient disclose relevant information when asked while avoiding unsupported or premature disclosure\.

The full patient construction achieves the best overall performance across both objective and subjective metrics\. This result suggests that faithful psychiatric simulation requires the combination of two complementary mechanisms: individualized symptom representation, which controls how the patient presents, and selective memory recall, which controls what evidence the patient discloses at each interaction step\. Without representation skill, the patient may remain factually grounded but lack clinical realism\. Without memory skill, the patient may appear expressive but become less controllable or more prone to unsupported disclosure\. Their combination allows the standardized patient to maintain EHR faithfulness while supporting natural, gradual, and clinically meaningful psychiatric interviews\.

## Appendix MFaithfulness Questionnaire

To further examine whether MentalHospital provides a clinically faithful and practically useful psychiatric simulation environment, we conduct a questionnaire\-based assessment with domain participants\. The study includes 5 psychiatrists, 3 psychologists, and 14 medical trainees\. Unlike objective comparison, which evaluates whether model outputs match EHR\-derived references, this questionnaire focuses on the perceived clinical plausibility, patient faithfulness, evaluation validity, and training utility of the overall environment\. This complementary assessment is necessary because a psychiatric simulation system should not only reproduce reference information, but also support interactions that are recognizable, interpretable, and useful from a clinical perspective\.

The questionnaire is organized into five dimensions: clinical realism \(CR\), patient fidelity \(PF\), patient distinctiveness \(PD\), evaluation credibility \(EC\), and training suitability \(TS\)\. Clinical realism assesses whether the interaction workflow resembles real psychiatric interviewing and clinical decision\-making, including the gradual progression from symptom inquiry to assessment, diagnosis, and treatment planning\. Patient fidelity evaluates whether the simulated patient remains consistent with the underlying psychiatric case information, especially in symptom expression, behavioral presentation, and response stability across turns\. Patient distinctiveness measures whether different simulated patients preserve meaningful individual differences rather than collapsing into homogeneous or generic response patterns\. Evaluation credibility assesses whether the dual\-track evaluation protocol captures clinically meaningful aspects of doctor\-agent performance\. Training suitability measures whether the system is useful for psychiatric education, interview practice, clinical reasoning training, and medical AI evaluation\.

Each item is rated on a 5\-point Likert scale, where higher scores indicate stronger agreement\. For each dimension, we compute the mean score across its corresponding items and participants\. We also report the overall mean score across all questionnaire items\. To summarize response polarity, we define scores of 4 or 5 as positive responses, scores of 1 or 2 as negative responses, and score 3 as neutral\. We report the positive and negative response rates over all collected ratings, together with 95% bootstrap confidence intervals\. This allows us to assess not only the average perceived quality of MentalHospital, but also the proportion of participants who explicitly endorse or reject its clinical faithfulness\.

Table[8](https://arxiv.org/html/2607.08257#A13.T8)lists the full questionnaire items\. The items are designed to cover both the construction quality of MentalHospital and its downstream utility\. The first three dimensions focus on whether the simulated clinical environment and patient agents are faithful to psychiatric practice and case\-specific evidence\. Specifically, clinical realism examines the plausibility of the overall clinical workflow, patient fidelity examines whether the AI patient remains grounded in the provided case, and patient distinctiveness examines whether the system can preserve case\-level heterogeneity\. The last two dimensions examine whether the evaluation protocol is clinically credible and whether the environment is suitable for training and assessment\.

In addition, the questionnaire is designed to separate fidelity at different levels\. Patient fidelity focuses on whether an individual simulated patient is consistent with the corresponding EHR\-derived case, whereas patient distinctiveness focuses on whether different cases lead to distinguishable patient behaviors and symptom profiles\. This distinction is important because a patient agent may appear realistic in isolation while still producing overly similar responses across cases\. Similarly, evaluation credibility is separated from training suitability: the former concerns whether the evaluation metrics and rubrics provide meaningful judgments of doctor\-agent performance, while the latter concerns whether the environment can support practice, feedback, and educational use\.

Before rating, participants are shown representative MentalHospital interactions and evaluation outputs, including patient responses, doctor\-agent inquiries, clinical notes, diagnostic reasoning, treatment planning, and corresponding evaluation results\. Participants are instructed to rate the system according to clinical plausibility and usefulness rather than surface fluency alone\. This instruction is intended to reduce the risk that high ratings are assigned merely because the generated text is coherent or polished\. Instead, the questionnaire emphasizes whether the simulated workflow, patient behavior, and evaluation design are clinically meaningful under psychiatric practice\.

The resulting questionnaire scores are used as an external validation of MentalHospital’s faithfulness\. While the objective metrics quantify whether doctor agents recover EHR\-derived evidence and produce correct task outputs, the questionnaire reflects whether domain participants perceive the environment as clinically plausible and useful\. Therefore, the faithfulness questionnaire provides a human\-centered complement to the benchmark’s quantitative evaluation protocol, supporting a broader assessment of whether MentalHospital can serve as a reliable environment for psychiatric interaction simulation, doctor\-agent evaluation, and training\-oriented use\.

Table 8:Expert questionnaire for evaluating the clinical fidelity of MentalHospital\.DimensionQuestionnaire ItemClinical RealismThe interaction workflow of MentalHospital is consistent with real\-world psychiatric interviewing practice\.Clinical RealismThe system supports a clinically plausible progression from symptom inquiry to assessment, diagnosis, and treatment planning\.Clinical RealismThe dialogue process reflects the sequential and exploratory nature of psychiatric evaluation\.Patient FidelityThe AI patient’s responses are consistent with the provided psychiatric case information\.Patient FidelityThe AI patient’s symptom expressions are clinically plausible for the corresponding psychiatric condition\.Patient FidelityThe AI patient maintains stable behavioral and emotional characteristics during the interaction\.Patient DistinctivenessDifferent AI patients exhibit distinguishable symptom profiles and response patterns\.Patient DistinctivenessThe simulated patients avoid homogeneous or generic responses across different cases\.Patient DistinctivenessThe system can reflect meaningful individual differences in patient presentation, history, and communication style\.Evaluation CredibilityThe objective evaluation metrics capture clinically relevant task outcomes in psychiatric assessment\.Evaluation CredibilityThe subjective assessment dimensions are appropriate for evaluating clinical interaction quality\.Evaluation CredibilityThe system’s evaluation results are interpretable and clinically meaningful\.Evaluation CredibilityThe combination of objective comparison and subjective assessment provides a credible evaluation of doctor\-agent performance\.Training SuitabilityMentalHospital is suitable for supporting psychiatric interview training\.Training SuitabilityMentalHospital can help trainees practice clinical reasoning in realistic psychiatric cases\.Training SuitabilityOverall, MentalHospital provides a clinically valuable environment for psychiatric education and medical AI evaluation\.
## Appendix NFrontend Workflow and Interaction Design ofMentalHospital

### N\.1Overview

This appendix describes the frontend workflow and interaction design ofMentalHospital\. The frontend is designed as the doctor\-facing interaction layer of an executable psychiatric training environment\. Rather than serving as a standalone web interface, it operationalizes the staged clinical workflow ofMentalHospitalby allowing medical students to enter cases, interview standardized patients, request auxiliary examinations, complete structured case writing, submit diagnostic hypotheses, formulate treatment plans, and review backend\-generated evaluation reports\.

The frontend plays an important role in preserving the sequential nature of psychiatric assessment\. In real clinical practice, information is not obtained all at once\. Clinicians gradually collect evidence through interview, examination, diagnostic reasoning, and treatment planning\. Following this principle, the frontend ofMentalHospitalorganizes user interaction into multiple stages and controls the information exposed to the user at each stage\. This design prevents the training episode from becoming a single unstructured prompt\-response exchange and instead supports a clinically grounded, stage\-aware interaction process\.

### N\.2Stage\-wise Interaction Workflow

The frontend workflow begins with a case\-selection interface\. After a case is selected, the user enters a sequential training episode consisting of four main clinical stages: patient interview, auxiliary examination, case writing and diagnosis, and treatment planning\. Each stage provides a specific interaction workspace and passes clinically relevant information to subsequent stages\.

##### Phase 1: Patient Interview\.

In the first phase, the frontend provides a multi\-turn interview interface in which the user acts as the doctor and interacts with a standardized psychiatric patient\. The patient responses are generated dynamically by the backend according to the selected case and the ongoing dialogue context\. The frontend records the complete dialogue trajectory and maintains the interview history throughout the episode\. This phase is intended to evaluate the user’s ability to collect psychiatric symptoms, explore illness history, assess mental status, and establish a clinically appropriate doctor\-patient interaction\.

##### Phase 2: Auxiliary Examination\.

After completing the interview, the frontend transitions to an auxiliary examination interface\. In this phase, the user selects examinations that may help clarify the patient’s condition\. The frontend presents available examination options and displays backend\-returned examination results according to the user’s selections\. Compared with the interview phase, this stage shifts the interaction from conversational information gathering to structured evidence acquisition\. The selected examinations and their results are preserved for later diagnostic reasoning and final evaluation\.

##### Phase 3: Case Writing and Diagnosis\.

The third phase combines the information obtained from the interview and auxiliary examinations into a structured case\-writing and diagnostic workspace\. The user is asked to complete a clinical case summary and then submit diagnostic hypotheses together with supporting reasons\. This phase is designed to assess whether the user can organize scattered clinical evidence into a coherent psychiatric case formulation and whether the proposed diagnosis is consistent with the collected information\.

Importantly, the frontend does not treat diagnosis as an isolated label\-selection task\. Instead, it provides an interaction space in which case writing, diagnostic selection, and diagnostic reasoning are connected\. This design reflects the clinical requirement that psychiatric diagnosis should be supported by symptom evidence, temporal course, differential considerations, and contextual information\.

##### Phase 4: Treatment Planning\.

In the fourth phase, the frontend presents a treatment\-planning interface\. The user is asked to formulate a treatment plan across multiple clinically meaningful categories, including pharmacological treatment, psychotherapy, neuromodulation or other somatic treatment, psychosocial intervention and rehabilitation, and auxiliary treatment\. This stage extends the training episode beyond diagnosis and requires the user to consider how diagnostic conclusions can be translated into an actionable intervention plan\.

By placing treatment planning after interview, examination, case writing, and diagnosis, the frontend preserves the intended clinical order of the system\. The final submitted treatment plan is stored together with previous\-stage outputs and becomes part of the complete episode record\.

### N\.3Backend\-coordinated Evaluation

After the user submits the treatment plan, the frontend enters a unified evaluation stage\. The frontend does not compute scores, generate judgments, or define evaluation outcomes locally\. Instead, it invokes backend evaluation endpoints and renders the returned results as an integrated report\. This design ensures that the evaluation process remains consistent with the backend evaluator and avoids inconsistencies caused by frontend\-side scoring logic\.

The unified evaluation report summarizes the user’s performance across the full training episode\. It covers multiple dimensions, including interview quality, auxiliary examination appropriateness, case\-writing quality, diagnostic correctness, diagnostic reasoning, and treatment\-plan completeness\. By collecting all stage\-level assessments into a single final report, the system provides a closed\-loop view of the user’s clinical performance rather than presenting fragmented feedback after each individual stage\.

This backend\-coordinated evaluation design also supports reproducibility\. Since the frontend only displays backend\-generated evaluation outputs, the scoring logic can be maintained, updated, and audited centrally on the backend side\. The frontend therefore functions as an execution and visualization layer, while the backend remains responsible for simulation, evaluation, and final result generation\.

### N\.4Episode Logging and Traceability

A key design goal of the frontend is to support complete episode traceability\. During the interaction process, the frontend collects and passes forward the dialogue history, selected auxiliary examinations, examination results, case\-writing content, diagnostic submissions, treatment plans, and final evaluation outputs\. These data are packaged as a complete training trajectory and stored by the backend\.

The history interface allows previously completed episodes to be revisited\. Each episode is treated as a full clinical training record rather than a temporary frontend session\. This design enables users and instructors to review how the interaction unfolded, what evidence was collected, how the diagnosis was formulated, what treatment plan was proposed, and how the final evaluation was generated\.

Such traceability is important for psychiatric training because errors may occur at different stages of the clinical reasoning process\. For example, an incorrect diagnosis may result from insufficient interview exploration, inappropriate examination selection, incomplete case writing, or flawed diagnostic reasoning\. By preserving the full trajectory,MentalHospitalmakes it possible to analyze not only the final answer but also the process that led to it\.

### N\.5Design Rationale

The frontend ofMentalHospitalis organized around three design principles\.

First, the frontend should preserve the staged structure of psychiatric assessment\. Each phase corresponds to a clinically meaningful step, and later stages depend on information collected earlier in the episode\. This structure encourages users to reason progressively rather than relying on one\-step answer generation\.

Second, the frontend should separate interaction control from evaluation authority\. The frontend is responsible for collecting user inputs, managing stage transitions, displaying backend outputs, and maintaining episode continuity\. Evaluation logic remains on the backend, which ensures consistency between the implemented system and the intended evaluation framework\.

Third, the frontend should support longitudinal review and training feedback\. By saving complete episode records, the system allows each training attempt to be reviewed as a traceable clinical reasoning trajectory\. This makes the frontend not only an interface for immediate interaction but also a component that supports later analysis, reflection, and educational feedback\.

### N\.6Extensibility

Because the frontend is organized around stage boundaries and backend\-coordinated operations, it can support new cases, additional evaluation dimensions, and future task extensions without changing the overall episode logic\. For example, new psychiatric case types can be introduced through backend case sampling, additional examination options can be added to the auxiliary examination stage, and new evaluation modules can be integrated into the unified report as long as the frontend preserves the same stage\-wise execution structure\.

This modular organization makes the frontend suitable for an evolving psychiatric training environment\. Its main contribution is not in providing a generic web application, but in offering a stable execution surface through which the proposed simulation, reasoning, evaluation, and logging mechanisms can be experienced as a complete interactive system\.

### N\.7Summary

In summary, the frontend ofMentalHospitalserves as the doctor\-facing execution layer of the proposed psychiatric training environment\. It supports staged interaction, structured information acquisition, backend\-coordinated evaluation, and complete episode traceability\. By connecting case entry, patient interview, auxiliary examination, case writing, diagnosis, treatment planning, final evaluation, and history review into a single workflow, the frontend helps instantiateMentalHospitalas an executable and traceable clinical training system\.