Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration

# Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human–LLM Collaboration
Source: [https://arxiv.org/html/2607.23538](https://arxiv.org/html/2607.23538)
Fatema Tuj Johora Faria1,Mukaffi Bin Moin1,Md\. Mahfuzur Rahman1,Khan Md Hasib2,3, Jubayer Al Mahmud4,M\. F\. Mridha5

1Ahsanullah University of Science and Technology,2Bangladesh University of Business and Technology, 3The University of New South Wales,4Jashore University of Science and Technology, 5American International University \- Bangladesh Correspondence:[ja\.mahmud@just\.edu\.bd](https://arxiv.org/html/2607.23538v1/mailto:[email protected]),[fatema\.faria142@gmail\.com](https://arxiv.org/html/2607.23538v1/mailto:[email protected]),[mukaffi28@gmail\.com](https://arxiv.org/html/2607.23538v1/mailto:[email protected])

###### Abstract

Despite recent advances in large language models \(LLMs\), their ability to generate empathetic mental health counseling responses in low\-resource languages remains largely unexplored\. To address this gap, we curate 625 authentic mental health cases from three complementary sources: \(1\) publicly available Facebook posts discussing mental health concerns, \(2\) transcripts from the Bangladeshi television program “Ami Akhon Ki Korbo”, and \(3\) anonymized student questionnaire responses covering diverse emotional and psychological challenges\. Based on these cases, we build an evaluation corpus comprising advice written by licensed clinical psychologists and responses generated by three modern proprietary LLMs: GPT\-4o Mini, Claude 4\.5 Haiku, and Gemini 2\.5 Pro\. We further propose theRole\-PlayingReflectiveChain\-of\-ThoughtAdvisoryFramework \(RP\-RCAF\), a task\-specific prompting strategy that combines expert\-authored few\-shot examples with structured self\-reflection to produce supportive, culturally aware, and ethically aligned counseling through a compassionate advisor persona\. We also introduce theGrok 4\-BasedResponseEvaluation andScoringFramework \(G\-REFS\), which integrates automated assessment with expert psychologist validation across emotional sensitivity, cultural appropriateness, linguistic clarity, and ethical soundness\. Experimental results show that RP\-RCAF consistently outperforms conventional prompting across all evaluated models and produces responses that more closely align with professional psychological counseling\. Disclaimer: This paper includes examples of sensitive mental health content intended solely for research and evaluation purposes\.

Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human–LLM Collaboration

Fatema Tuj Johora Faria1, Mukaffi Bin Moin1, Md\. Mahfuzur Rahman1, Khan Md Hasib2,3,Jubayer Al Mahmud4,M\. F\. Mridha51Ahsanullah University of Science and Technology,2Bangladesh University of Business and Technology,3The University of New South Wales,4Jashore University of Science and Technology,5American International University \- BangladeshCorrespondence:[ja\.mahmud@just\.edu\.bd](https://arxiv.org/html/2607.23538v1/mailto:[email protected]),[fatema\.faria142@gmail\.com](https://arxiv.org/html/2607.23538v1/mailto:[email protected]),[mukaffi28@gmail\.com](https://arxiv.org/html/2607.23538v1/mailto:[email protected])

## 1Introduction

Mental health is a fundamental pillar of human well\-being, shaping the way individuals think, feel, act, and connect with others, yet mental health challenges remain deeply pervasive and often overlooked, especially in fast\-paced and interconnected modern societies\. The pressures of daily life, from personal responsibilities to professional expectations, can contribute to emotional strain, while the time between therapeutic sessions may become a vulnerable period marked by loneliness, self\-doubt, and emotional instability\. Unfortunately, the availability of timely, affordable, and continuous psychological support remains limited due to financial barriers, complex appointment systems, and a shortage of trained mental health professionalsEshghie and Eshghie \([2023](https://arxiv.org/html/2607.23538#bib.bib1)\)Liuet al\.\([2023](https://arxiv.org/html/2607.23538#bib.bib2)\)\.

A survey by the National Institute of Mental Health \(2018–2019\) reports that about 17% of adults in Bangladesh suffer from mild to severe mental health conditions, with higher prevalence among women \(19%\) than men \(15%\)\. These figures underscore not only a vast treatment gap but also a systemic failure to prioritize emotional and psychological well\-being within national public health strategies111[https://www\.who\.int/bangladesh/health\-topics/mental\-health](https://www.who.int/bangladesh/health-topics/mental-health)\.

In recent years, large language models \(LLMs\) such as Gemini 2\.5 Pro, LLaMA 4, DeepSeek R1, and Qwen2\.5\-Max have demonstrated strong capabilities in generating human\-like, context\-aware responses across a wide range of domains\. As a result, these models are increasingly used in English\-language settings for mental health applications, such as initial well\-being assessment, stress management, and guided cognitive behavioral exercises\. By enabling emotionally intelligent and context\-sensitive interactions, LLMs provide scalable and accessible first\-line mental health support, particularly in contexts where professional care is limited or costlyKanget al\.\([2025](https://arxiv.org/html/2607.23538#bib.bib13)\); Aggarwalet al\.\([2025](https://arxiv.org/html/2607.23538#bib.bib14)\); Neupaneet al\.\([2025](https://arxiv.org/html/2607.23538#bib.bib15)\); Na \([2024](https://arxiv.org/html/2607.23538#bib.bib16)\)\.

Despite some progress in Bangla Natural Language Processing \(NLP\) on tasks such as sentiment analysisKabiret al\.\([2023b](https://arxiv.org/html/2607.23538#bib.bib20)\), hate speech detectionFariaet al\.\([2024](https://arxiv.org/html/2607.23538#bib.bib18)\), emotion classificationKabiret al\.\([2023a](https://arxiv.org/html/2607.23538#bib.bib19)\), toxicity detectionMoinet al\.\([2024](https://arxiv.org/html/2607.23538#bib.bib17)\), suicidal ideation identificationPaulet al\.\([2024](https://arxiv.org/html/2607.23538#bib.bib22)\), and depression detectionChowdhuryet al\.\([2024](https://arxiv.org/html/2607.23538#bib.bib21)\), mental health NLP in Bangla is still in its early stages\. Importantly, in the Bangladeshi context, prior research has predominantly focused on mental health detection tasks, with little to no attention given to response generation or counseling support using LLMs\. This study aims to investigate key research questions on AI\-assisted mental health support in Bangladesh\.

1. RQ1\.How effective are LLMs in generating culturally sensitive and empathetic mental health advice in Bangla compared to responses crafted by licensed human psychologists in the Bangladeshi context?
2. RQ2\.To what extent does the proposed RP\-RCAF enhance the ethical soundness and contextual appropriateness of LLM\-generated mental health responses for Bangladeshi users?
3. RQ3\.How does human\-in\-the\-loop validation influence the refinement of LLM\-generated mental health advice to ensure psychological safety for Bangladeshi populations?
4. RQ4\.To what extent do G\-REFS evaluations agree with human expert assessments in evaluating the quality of generated mental health advice within the Bangladeshi socio\-cultural context?

![Refer to caption](https://arxiv.org/html/2607.23538v1/x1.png)

Figure 1:Overview of the Bangla mental health counseling case study\.\(1\)Authentic counseling cases are collected from three complementary sources\.\(2\)The cases are anonymized, preprocessed, and categorized into mental health topics\.\(3\)Counseling responses are developed by licensed clinical psychologists and generated by proprietary LLMs using the proposed RP\-RCAF framework\.\(4\)All responses are evaluated using G\-REFS with expert psychologist validation to construct the final evaluation corpus\.
## 2Related Works

### 2\.1Current Developments in Bangla Mental Health Support Systems

Existing studies on mental health support for Bangla speakers have primarily focused on detecting issues through social media and text analysis, without offering supportive or culturally appropriate responses\. For example, Opinion\-BERTHossainet al\.\([2025](https://arxiv.org/html/2607.23538#bib.bib23)\)effectively classifies emotions in Bangla text using BERT, CNN, and BiGRU but lacks mechanisms for providing helpful advice\. Another studyChowdhuryet al\.\([2024](https://arxiv.org/html/2607.23538#bib.bib21)\)used models, such as GPT\-3\.5, GPT\-4, and DepGPT to identify depression in Bangla social media posts and introduced the BSMDD dataset; however, it centers solely on detection\. Similarly, a study employing machine learning techniques to identify suicidal postsPaulet al\.\([2024](https://arxiv.org/html/2607.23538#bib.bib22)\)achieves strong classification performance but does not address response generation\. Although these efforts advance detection, they overlook the need for empathetic and actionable support\. Our work with theMindSpeak\-Bangladataset addresses this gap by integrating human expertise and LLMs to deliver compassionate, context\-aware responses tailored for Bangla speakers in real\-world counseling settings\.

### 2\.2Global Perspectives on Mental Health Support Approaches

Previous studies have explored the use of LLMs to enhance mental health support across various languages and regions, such as EnglishEshghie and Eshghie \([2023](https://arxiv.org/html/2607.23538#bib.bib1)\), ChineseSunet al\.\([2021](https://arxiv.org/html/2607.23538#bib.bib24)\), SpanishMármol\-Romeroet al\.\([2025](https://arxiv.org/html/2607.23538#bib.bib25)\), and HindiAgarwal and Abbas \([2025](https://arxiv.org/html/2607.23538#bib.bib28)\)\. These efforts aim to improve accessibility and broaden the reach of mental health care through chatbots and AI\-driven systems\. For example, one studyEshghie and Eshghie \([2023](https://arxiv.org/html/2607.23538#bib.bib1)\)uses ChatGPT as a therapy assistant to gather patient information, engage in supportive dialogue, and generate summaries for therapists while avoiding medical advice\. Building on this, the PsyQA datasetSunet al\.\([2021](https://arxiv.org/html/2607.23538#bib.bib24)\)offers thousands of Chinese mental health questions and answers, some integrating counseling techniques rooted in therapeutic principles\. In Spanish\-speaking regions, another projectMármol\-Romeroet al\.\([2025](https://arxiv.org/html/2607.23538#bib.bib25)\)developed a Telegram chatbot for adolescents aged 12 to 18 years to promote mental health awareness and emotional expression using culturally relevant GPT\-based responses\. Likewise, a Hindi\-language studyAgarwal and Abbas \([2025](https://arxiv.org/html/2607.23538#bib.bib28)\)used LLMs to generate synthetic emotional datasets, thereby improving emotion classification for rare mental states\. However, many of these approaches lack human\-in\-the\-loop \(HITL\) validation and automated grading of responses, potentially limiting the accuracy and depth of their support, and none adequately addresses the needs of the Bangla language\.

## 3Case Study Design

We conduct a case study to evaluate the effectiveness of LLMs in generating mental health counseling responses in Bangla\. The study uses authentic counseling cases collected from diverse sources in Bangladesh\. Figure[1](https://arxiv.org/html/2607.23538#S1.F1)illustrates the overall workflow, including case collection, preprocessing, and counseling response generation\. Comprehensive details of the data collection sources and preprocessing pipeline are provided in Appendix[A](https://arxiv.org/html/2607.23538#A1)\.

### 3\.1Counseling Response Development

#### 3\.1\.1Expert Counseling Responses

To establish reliable reference responses for the case study, licensed clinical psychologists developed professional counseling advice for a subset of the collected mental health cases\. Approximately 30% of the curated cases were selected to serve as expert references for both qualitative comparison and few\-shot prompting\.

- •Phase 1\) Case Selection:All collected cases underwent a rigorous anonymization process to remove personally identifiable information before further analysis\. The anonymized cases were then systematically organized into major mental health categories frequently observed in the Bangladeshi context, such as academic stress, interpersonal conflict, loneliness, anxiety, depression, family problems, and self\-harm ideation\. This categorization ensured broad coverage of diverse, real\-world counseling scenarios\.
- •Phase 2\) Expert Assignment:The selected cases were independently assigned to two licensed clinical psychologists with extensive professional experience in providing mental health counseling within the Bangladeshi sociocultural context\. To preserve privacy, the identities of both experts remain confidential\.
- •Phase 3\) Expert Response Development:Each psychologist prepared counseling responses in natural and compassionate Bangla\. Rather than providing clinical diagnoses or medical recommendations, the responses emphasized empathetic communication, emotional validation, practical coping strategies, and appropriate encouragement to seek professional support whenever necessary\. The experts also maintained adherence to established ethical principles for mental health counseling in every response\.
- •Phase 4\) Expert Consensus Review:After the initial response development, both psychologists independently evaluated the counseling responses, with inter\-rater agreement measured using Cohen’s KappaBlackman and Koval \([2000](https://arxiv.org/html/2607.23538#bib.bib7)\)\(κ=0\.83\\kappa=0\.83\)\. Disagreements were resolved through discussion, and overly prescriptive, ambiguous, or potentially harmful responses were revised\. The finalized expert responses served as the reference standard and were incorporated as few\-shot demonstrations within the proposed RP\-RCAF prompting framework\.

#### 3\.1\.2LLM Counseling Response Generation

To evaluate the capability of response generation in mental health counseling, approximately 70% of the collected cases were assigned to three modern LLMs: GPT\-4o MiniOpenAI \([2024](https://arxiv.org/html/2607.23538#bib.bib35)\), Claude 4\.5 HaikuAnthropic \([2025](https://arxiv.org/html/2607.23538#bib.bib33)\), and Gemini 2\.5 ProGoogle \([2025](https://arxiv.org/html/2607.23538#bib.bib34)\)\. Rather than relying on conventional prompting, we employed the proposed RP\-RCAF framework to guide response generation\. The framework integrates expert\-authored few\-shot demonstrations with structured reflective reasoning, enabling the models to generate counseling responses that are compassionate, culturally appropriate, and ethically responsible\.

For every counseling case, RP\-RCAF instructed the models to satisfy three fundamental objectives:

1. i\)Empathetic and culturally grounded counseling:The framework encouraged the models to acknowledge users’ emotional states with warmth and compassion while considering the sociocultural characteristics of Bangladesh, such as cultural traditions, religious considerations, interpersonal relationships, societal expectations, and contextual lived experiences\. Responses were expected to reflect these contextual factors so that the guidance remained authentic, respectful, accessible, and personally relevant\.
2. ii\)Ethically responsible guidance:RP\-RCAF explicitly discouraged the models from providing clinical diagnoses, prescribing medication, or making unsupported medical recommendations\. Instead, the models focused on emotional validation, constructive coping strategies, and appropriate encouragement to seek professional mental health support whenever severe psychological distress or safety risks were identified\.
3. iii\)Actionable counseling support:Beyond emotional reassurance, the framework instructed the models to provide practical and supportive advice that users could realistically apply in their daily lives\. This included stress management techniques, mindfulness practices, journaling, healthy communication strategies, and recommendations for recognizing situations that require professional intervention\.

### 3\.2Case Study Statistics

The evaluation corpus consists of 625 authentic mental health cases: 188 expert\-authored counseling responses and 437 responses generated by proprietary LLMs after quality filtering\. Each case contains the mental health category, the original user concern in Bangla, the corresponding counseling response, and the source of the query\. Responses are further annotated by author type \(human or LLM\), with the specific LLM identified when applicable\. Figure[3](https://arxiv.org/html/2607.23538#A1.F3)presents representative counseling cases, while Figure[7](https://arxiv.org/html/2607.23538#A3.F7)summarizes the distribution of the case study\.

![Refer to caption](https://arxiv.org/html/2607.23538v1/x2.png)

Figure 2:Overview of the proposed RP\-RCAF and G\-REFS frameworks\.\(1\)RP\-RCAF employs role\-playing and reflective advisory prompting to generate compassionate, culturally grounded, and ethically responsible counseling responses\.\(2\)G\-REFS combines automated evaluation with validation by human experts to assess response quality\.

## 4Implementation Details

### 4\.1Role\-Playing Reflective Chain\-of\-Thought Advisory Framework

TheRP\-RCAFis afew\-shot chain\-of\-thought \(CoT\)prompting framework that instructs the model to assume the role of a compassionate and culturally aware mental health advisor\. The model is guided using carefully selected few\-shot examples of responses written by licensed human experts, which serve as gold\-standard references that help the model internalize the appropriate empathetic tone, cultural sensitivity, and ethical constraints necessary for providing contextually relevant and emotionally supportive advice\.

Formally, given a user queryQQ, the system generates a responseRRthrough intermediate reasoning steps\{r1,r2,…,rk\}\\\{r\_\{1\},r\_\{2\},\\ldots,r\_\{k\}\\\}, which guide the final output\. This CoT prompting encourages the model to decompose complex queries into smaller components before producing a coherent response\.

R=𝒢​\(Q,T,S,\{ri\}i=1k\)R=\\mathcal\{G\}\\big\(Q,T,S,\\\{r\_\{i\}\\\}\_\{i=1\}^\{k\}\\big\)
where:

- •QQis the input user query text,
- •T∈\{t1,t2,…,tm\}T\\in\\\{t\_\{1\},t\_\{2\},\\ldots,t\_\{m\}\\\}is the inferred mental health topic category \(e\.g\., sexual abuse, breakup\),
- •SSrepresents safety and ethical constraints imposed during generation,
- •\{ri\}i=1k\\\{r\_\{i\}\\\}\_\{i=1\}^\{k\}are the intermediate reflective reasoning steps generated as part of the chain\-of\-thought,
- •𝒢\\mathcal\{G\}is the generation function operationalized through prompting LLMs\.

### 4\.2Multi\-Model Querying and Response Variants Generation

For each anonymized user query, the RP\-RCAF template was applied to all models to generate candidate responses\. We experimented with different parameter settings to balance creativity and reliability in the generated outputs\. After extensive tuning, we selected a standard configuration of temperature 0\.7 and top\-p 0\.9 for all models\. All candidate responses generated for each query were collected and stored in a centralized repository for expert evaluation\.

### 4\.3Response Evaluation and Scoring Framework

To ensure that every response in our dataset met high standards of emotional support, ethical responsibility, and cultural appropriateness, we developed theGrok 4\-Based Response Evaluation and Scoring Framework \(G\-REFS\)\. In this framework, Grok 4xAI \([2025](https://arxiv.org/html/2607.23538#bib.bib36)\)serves as theLLM judgeto systematically assess the quality of responses generated by the three primary models\.

#### 4\.3\.1Theory\-Driven Evaluation Criteria

To systematically evaluate generated advice, we adopt a theory\-driven framework in which each criterion represents a key dimension of effective mental health support and is measured using a 5\-point Likert scaleJoshiet al\.\([2015](https://arxiv.org/html/2607.23538#bib.bib32)\)\. We denote each dimension asTiT\_\{i\}, whereiiindexes a specific construct\. EachTiT\_\{i\}is independently scored by theLLM judgeon a scale from 1 \(lowest\) to 5 \(highest\) based on predefined descriptors, which ensures consistent and quantitative evaluation\. Detailed definitions ofT1T\_\{1\}–T4T\_\{4\}are provided in Appendix[B](https://arxiv.org/html/2607.23538#A2)\.

#### 4\.3\.2Score\-Based Categorization

After obtaining the four individual scores for each responseRj\(i\)R\_\{j\}^\{\(i\)\}, we computed the average scores¯\\bar\{s\}as follows:

s¯=14​∑k=14sk,\\bar\{s\}=\\frac\{1\}\{4\}\\sum\_\{k=1\}^\{4\}s\_\{k\},
wheresks\_\{k\}denotes the score assigned to thekk\-th evaluation criterion\.

The categorization thresholds follow the semantic interpretation of the 5\-point Likert scale\. Responses with an average score of at least 4 are accepted directly\. Responses with average scores between 2\.5 and 4 meet some evaluation criteria but require expert\-guided revision\. Responses with average scores below 2\.5 are rejected and regenerated due to deficiencies across multiple evaluation dimensions\. This three\-tier categorization provides a systematic quality\-control mechanism while reserving human review for responses that require refinement\.

Based on the average score, each response was assigned to one of the following categories:

- •Accept:If the response satisfies the predefined quality criteria and is accepted without further modification\.
- •Revise:If 2\.5≤s¯<4,2\.5\\leq\\bar\{s\}<4,The response requires expert review and refinement to improve aspects such as tone, clarity, cultural sensitivity, or ethical alignment\. The revised response is subsequently re\-evaluated by theLLM judge, and responses that satisfy the acceptance criterion are retained in the final dataset\.
- •Reject:If the response exhibits substantial deficiencies in empathy, factual reliability, and ethical safety and is excluded from the dataset\. The corresponding query is then submitted to the generating LLM to produce a new response, which subsequently undergoes the same evaluation procedure\.

#### 4\.3\.3Human\-in\-the\-Loop Validation

After the initial automated scoring by theLLM judge, responses classified asAcceptwere further reviewed by two licensed clinical psychologists, while responses classified asReviseunderwent expert\-guided editing by trained annotators under their supervision\. The revised responses were then re\-evaluated by theLLM judge, and those that achieved anAcceptrating were included in the dataset to support future research\.

## 5Results Analysis

### 5\.1Model Performance Across Mental Health Categories

Tables[1](https://arxiv.org/html/2607.23538#S5.T1)and[2](https://arxiv.org/html/2607.23538#S5.T2)show the average G\-REFS scores for human and LLM responses across the eight mental health categories, along with the proportions of responses initially accepted \(s¯≥4\\bar\{s\}\\geq 4\) and after revision\.

Findings for RQ1As summarized in Table[1](https://arxiv.org/html/2607.23538#S5.T1)and Table[2](https://arxiv.org/html/2607.23538#S5.T2), the evaluated proprietary LLMs exhibit a consistent performance ranking, with Gemini 2\.5 Pro achieving the highest overall scores, followed by Claude 4\.5 Haiku and GPT\-4o Mini\. Across all models,T3T\_\{3\}receives the highest average scores, followed byT1T\_\{1\}andT4T\_\{4\}, whereasT2T\_\{2\}consistently remains the weakest dimension, highlighting persistent challenges in culturally grounded response generation\. Compared with licensed psychologists, proprietary LLMs achieve competitive performance across most evaluation dimensions but consistently obtain lower scores, particularly inT2T\_\{2\}and in highly sensitive counseling scenarios involving sexual abuse and self\-destructive thoughts, where contextual interpretation and human judgment are especially critical\. These findings indicate that while current LLMs provide effective assistance for Bangla mental health counseling, expert human involvement remains essential for addressing culturally nuanced and psychologically complex situations while maintaining dependable, ethically aligned, and context\-sensitive counseling practices\.

### 5\.2Impact of the RP\-RCAF Framework

Table 1:Average G\-REFS scores for human\-generated counseling responses across different mental health categories\.Table 2:Performance of different LLMs across mental health categories, measured by average G\-REFS scores under the proposed RP\-RCAF framework\.Table 3:Comparison of zero\-shot \(ZS\) prompting, few\-shot \(FS\) prompting, and the proposed RP\-RCAF across different LLMs using average G\-REFS scores\. Improvement \(%\) denotes the relative gain of RP\-RCAF over the corresponding baseline \(ZS or FS\), computed asRP\-RCAF−BaselineBaseline×100\\frac\{\\text\{RP\-RCAF\}\-\\text\{Baseline\}\}\{\\text\{Baseline\}\}\\times 100\.The expert\-curated examples inRP\-RCAFprovide high\-quality references for empathy, tone, and response style, while structured reflective reasoning guides the model to consider the user’s emotional state and contextual factors before generating advice\. As a result,RP\-RCAFproduces more empathetic, culturally appropriate, and context\-aware responses than conventional approaches\. Table[3](https://arxiv.org/html/2607.23538#S5.T3)summarizes the performance of zero\-shot \(ZS\), few\-shot \(FS\), andRP\-RCAFunder the four theory\-driven evaluation criteria \(T1T\_\{1\}–T4T\_\{4\}\)\.

### 5\.3Impact of Human\-in\-the\-Loop Validation

Licensed clinical psychologists play a critical role in refining LLM responses categorized asRevise\(2\.5≤s¯<42\.5\\leq\\bar\{s\}<4\), improving their therapeutic quality, cultural appropriateness, and ethical reliability before deployment\. Table[5](https://arxiv.org/html/2607.23538#A3.T5)reports the proportion of responses requiring revision and the success rate of revised responses achievings¯≥4\\bar\{s\}\\geq 4\.

### 5\.4Agreement Between G\-REFS and Human Expert Evaluations

We measured the agreement between G\-REFS automated evaluations and independent psychologist ratings using theIntraclass Correlation Coefficient \(ICC\)Koo and Li \([2016](https://arxiv.org/html/2607.23538#bib.bib8)\), which quantifies the consistency between two sets of ratings\. Table[6](https://arxiv.org/html/2607.23538#A3.T6)summarizes the category\-wise and overall ICC values across the four evaluation dimensions \(𝐓1\\mathbf\{T\}\_\{1\},𝐓2\\mathbf\{T\}\_\{2\},𝐓3\\mathbf\{T\}\_\{3\}, and𝐓4\\mathbf\{T\}\_\{4\}\)\.

Findings for RQ2Table[3](https://arxiv.org/html/2607.23538#S5.T3)demonstrates that the proposed RP\-RCAF consistently outperforms both zero\-shot and few\-shot approaches across all evaluated LLMs \(RP\-RCAF\>\>FS\>\>ZS\), which confirms the effectiveness of the framework across diverse model families\. Compared with the baseline strategies, RP\-RCAF yields the largest improvements in the human\-centered evaluation dimensions, particularly𝐓1\\mathbf\{T\}\_\{1\}and𝐓2\\mathbf\{T\}\_\{2\}, although the relative gains vary across models\. The framework also achieves substantial gains in𝐓4\\mathbf\{T\}\_\{4\}, which indicates stronger ethical alignment and safer counseling responses\. In contrast, improvements in𝐓3\\mathbf\{T\}\_\{3\}remain modest because all baseline models already achieve high linguistic clarity, with less room for improvement\. In summary, these findings show that RP\-RCAF improves the quality of mental health counseling responses beyond conventional approaches\.

Findings for RQ3Table[5](https://arxiv.org/html/2607.23538#A3.T5)shows that human\-in\-the\-loop validation substantially improved the overall quality of RP\-RCAF\-generated mental health responses across all evaluated LLMs\. Expert review refined responses by strengthening emotional validation, replacing overly directive or clinical language with more compassionate and non\-judgmental expressions, and introducing culturally relevant coping strategies tailored to the Bangladeshi context\. These refinements proved especially valuable for high\-risk scenarios, such as self\-destructive thoughts and sexual abuse, where greater sensitivity, contextual understanding, and careful judgment were required\. Overall, the findings demonstrate that integrating RP\-RCAF with expert human oversight yields more reliable, ethically sound, and context\-aware counseling responses for Bangla\-speaking users\.

Findings for RQ4Table[6](https://arxiv.org/html/2607.23538#A3.T6)indicates moderate\-to\-good agreement between G\-REFS and independent psychologist evaluations \(overall ICC = 0\.71\), following standard interpretation guidelines\. Across the four evaluation dimensions,𝐓3\\mathbf\{T\}\_\{3\}achieves the highest agreement, whereas𝐓2\\mathbf\{T\}\_\{2\}and𝐓4\\mathbf\{T\}\_\{4\}show comparatively lower consistency, reflecting the inherent difficulty of automatically assessing cultural context and ethical considerations\. Notably, agreement is lowest for the highest\-risk categories, Self\-Destruction \(ICC = 0\.57\) and Sexual Abuse \(ICC = 0\.59\), suggesting that G\-REFS is less dependable in scenarios where assessment accuracy is most critical\. In contrast, categories such as Loneliness \(0\.75\) and Depression \(0\.74\) exhibit stronger agreement\. Overall, the results suggest that G\-REFS offers a reliable automated evaluation signal for lower\-risk counseling scenarios, whereas culturally sensitive and high\-risk cases still benefit from expert human oversight to maintain accurate interpretation and psychologically safe responses\.

## 6Future Work

Future work will focus on developing LLM agents as virtual mental health companions capable of providing personalized, context\-aware counseling\. These agents will incorporate cognitive modules for goal\-oriented planning, long\-term memory to maintain conversational context, tool integration for crisis intervention, and multi\-agent collaboration with human moderators when necessary\. The overall framework will be designed in accordance with the WHOmhGAP Intervention Guideand culturally informed counseling practices in Bangladesh\. To improve inclusivity, theMindSpeak\-Banglaevaluation corpus will be expanded to include counseling cases from underrepresented communities, including rural populations, low\-income groups, and persons with disabilities\. We also plan to incorporate multilingual and code\-switched conversations and explore language\-aware LLM agents that automatically adapt their responses to users’ preferred languages and dialects\.

## 7Conclusion

In this paper, we conducted a comprehensive investigation of Bangla mental health response generation using authentic scenarios drawn from the Bangladeshi sociocultural context\. We introducedMindSpeak\-Bangla, a curated corpus of 625 real\-world cases paired with advice from licensed clinical psychologists and three proprietary LLMs\. We examined the ability of LLMs to provide support for sensitive mental health cases, evaluated the proposedRP\-RCAF, and assessed output quality through theG\-REFSframework with validation by licensed clinical psychologists\. Experimental results showed that, across all evaluated models, the proposedRP\-RCAFconsistently outperformed conventional prompting strategies \(RP\-RCAF≻\\succFS≻\\succZS\)\. Among the evaluated LLMs, Gemini 2\.5 Pro achieved the strongest overall performance, followed by Claude 4\.5 Haiku and GPT\-4o Mini, while expert\-authored responses remained the highest\-performing overall\. The findings showed that effective counseling depends not only on linguistic ability but also on cultural and situational understanding\. This work establishes a strong foundation for future research on culturally grounded AI\-assisted mental health support and promotes the development and evaluation of LLMs aligned with the sociocultural contexts in which they are deployed\.

## Limitations

This work has several limitations\. First, the study focuses on nonclinical mental health counseling and is intended to provide emotional support rather than clinical diagnosis or therapeutic intervention\. Although the expert\-authored responses were written by licensed clinical psychologists, LLM\-generated responses may not consistently reflect the depth of professional counseling expertise\. Second, the evaluation corpus is derived from Facebook posts, television program transcripts, and university student questionnaires, which may not fully represent the diversity of mental health concerns across different demographic and socioeconomic groups in Bangladesh\. Third, the proposed RP\-RCAF relies on prompt engineering, making response quality sensitive to prompt design and limiting reproducibility across different prompting strategies\. Furthermore, despite careful anonymization and expert review, ethical and safety risks remain when addressing highly sensitive topics such as self\-harm and sexual abuse\. The current case study also considers only single\-turn counseling scenarios, without modeling the long\-term conversational context required in real\-world mental health support\. Finally, the study evaluates only proprietary LLMs, and the findings may not directly generalize to open\-source or future foundation models\.

## Ethics Statement

This work presents a case study on Bangla mental health counseling using authentic counseling scenarios collected from the Bangladeshi context\. Owing to the sensitive nature of mental health data, we followed established ethical principles throughout data collection, preprocessing, response generation, evaluation, and resource release\.

##### Data Collection Ethics\.

The evaluation corpus was constructed from three sources: publicly available Facebook group posts, transcripts from a publicly broadcast television program, and voluntary university student questionnaires\. Only publicly accessible Facebook content was collected without bypassing privacy settings or membership restrictions\. We recognize that public availability does not imply informed consent for research use\. Student questionnaire responses were collected voluntarily under informed consent and without academic, financial, or institutional coercion\. Participants were informed about the purpose of the study, allowed to skip any question they were uncomfortable answering, and were not provided with counseling or clinical intervention through the questionnaire\. The collected responses were used solely for research purposes and were anonymized before analysis\.

##### Privacy and Anonymization\.

All counseling cases underwent rigorous anonymization before response generation and evaluation\. Personally identifiable information, including names, contact information, locations, and other sensitive identifiers, was removed\. Given the sensitive nature of mental health narratives, we treat re\-identification as a potential risk and release only fully anonymized cases\. Examples presented in the paper are paraphrased where necessary to provide additional privacy protection while preserving their contextual meaning\.

##### Human Oversight\.

To promote response quality and safety, all LLM\-generated counseling responses were assessed using the proposed G\-REFS framework and independently reviewed by licensed clinical psychologists\. Responses that failed to satisfy safety, ethical, or cultural standards were revised or excluded\. Although these safeguards substantially reduce potential risks, they cannot completely eliminate inappropriate or harmful outputs under every possible counseling scenario\.

##### Fairness and Representativeness\.

The evaluation corpus reflects mental health concerns within the Bangladeshi sociocultural context; however, its sources do not fully represent all demographic groups\. Rural communities, ethnic minorities, older adults, and other underrepresented populations remain limited\. Consequently, the findings of this study should not be generalized beyond the represented population without careful judgment\.

##### Use of Proprietary LLMs\.

Counseling responses were generated using proprietary LLMs through their official APIs\. No personally identifiable or non\-anonymized information was submitted to these services, and all usage complied with the respective provider policies\.

##### Resource Release\.

TheMindSpeak\-Banglaevaluation corpus will be released exclusively for non\-commercial research purposes\. Accompanying documentation describes the intended use, known limitations, and ethical considerations\. We encourage responsible use and recommend appropriate human oversight for any downstream mental health applications\.

## References

- Emotion detection in hindi language using gpt and bert\.InArtificial Intelligence XLI,M\. Bramer and F\. Stahl \(Eds\.\),Cham,pp\. 105–118\.External Links:ISBN 978\-3\-031\-77918\-3,[Link](https://doi.org/10.1007/978-3-031-77918-3_8)Cited by:[§2\.2](https://arxiv.org/html/2607.23538#S2.SS2.p1.1)\.
- V\. Aggarwal, S\. Thukral, K\. Patel, and A\. Chatterjee \(2025\)Leveraging llms for mental health: detection and recommendations from social discussions\.External Links:2503\.01442,[Link](https://arxiv.org/abs/2503.01442)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p3.1)\.
- Anthropic \(2025\)Claude 4\.5 haiku\.Note:[https://www\.anthropic\.com/claude](https://www.anthropic.com/claude)Accessed: 31 January 2026Cited by:[§3\.1\.2](https://arxiv.org/html/2607.23538#S3.SS1.SSS2.p1.1)\.
- N\. J\. Blackman and J\. J\. Koval \(2000\)Interval estimation for cohen’s kappa as a measure of agreement\.Statistics in medicine19\(5\),pp\. 723–741\.Cited by:[4th item](https://arxiv.org/html/2607.23538#S3.I1.i4.p1.1)\.
- A\. K\. Chowdhury, S\. R\. Sujon, M\. S\. S\. Shafi, T\. Ahmmad, S\. Ahmed, K\. M\. Hasib, and F\. M\. Shah \(2024\)Harnessing large language models over transformer models for detecting bengali depressive social media text: a comprehensive study\.Natural Language Processing Journal7,pp\. 100075\.Cited by:[Table 4](https://arxiv.org/html/2607.23538#A2.T4.9.1.4.3.1),[§1](https://arxiv.org/html/2607.23538#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.23538#S2.SS1.p1.1)\.
- M\. Eshghie and M\. Eshghie \(2023\)ChatGPT as a therapist assistant: a suitability study\.External Links:2304\.09873,[Link](https://arxiv.org/abs/2304.09873)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.23538#S2.SS2.p1.1)\.
- F\. T\. J\. Faria, L\. H\. Baniata, and S\. Kang \(2024\)Investigating the predominance of large language models in low\-resource bangla language over transformer models for hate speech detection: a comparative analysis\.Mathematics12\(23\),pp\. 3687\.External Links:[Document](https://dx.doi.org/10.3390/math12233687),[Link](https://doi.org/10.3390/math12233687)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p4.1)\.
- Gamitisa \(2026\)Banglish to bangla converter\.Note:Online tool, accessed: 31 July 2025\. Available at: https://gamitisa\.com/tools/banglish\-to\-banglaCited by:[§A\.1](https://arxiv.org/html/2607.23538#A1.SS1.p3.1)\.
- Google \(2025\)Gemini 2\.5 pro \(preview version, as of 31 july 2025\) \[large multimodal model accessed via google ai studio\]\.Note:[https://aistudio\.google\.com/](https://aistudio.google.com/)Accessed: 05 February 2026Cited by:[§3\.1\.2](https://arxiv.org/html/2607.23538#S3.SS1.SSS2.p1.1)\.
- Md\. M\. Hossain, Md\. S\. Hossain, Md\. F\. Mridha,et al\.\(2025\)Multi task opinion enhanced hybrid bert model for mental health analysis\.Scientific Reports15,pp\. 3332\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-86124-6),[Link](https://doi.org/10.1038/s41598-025-86124-6)Cited by:[§2\.1](https://arxiv.org/html/2607.23538#S2.SS1.p1.1)\.
- A\. Joshi, S\. Kale, S\. Chandel, and D\. Pal \(2015\)Likert scale: explored and explained\.British Journal of Applied Science & Technology7,pp\. 396–403\.External Links:[Document](https://dx.doi.org/10.9734/BJAST/2015/14975)Cited by:[§4\.3\.1](https://arxiv.org/html/2607.23538#S4.SS3.SSS1.p1.5)\.
- A\. Kabir, A\. Roy, and Z\. Taheri \(2023a\)BEmoLexBERT: a hybrid model for multilabel textual emotion classification in Bangla by combining transformers with lexicon features\.InProceedings of the First Workshop on Bangla Language Processing \(BLP\-2023\),F\. Alam, S\. Kar, S\. A\. Chowdhury, F\. Sadeque, and R\. Amin \(Eds\.\),Singapore,pp\. 56–61\.External Links:[Link](https://aclanthology.org/2023.banglalp-1.7/),[Document](https://dx.doi.org/10.18653/v1/2023.banglalp-1.7)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p4.1)\.
- M\. Kabir, O\. Bin Mahfuz, S\. R\. Raiyan, H\. Mahmud, and M\. K\. Hasan \(2023b\)BanglaBook: a large\-scale Bangla dataset for sentiment analysis from book reviews\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 1237–1247\.External Links:[Link](https://aclanthology.org/2023.findings-acl.80/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.80)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p4.1)\.
- M\. K\. Kabir, M\. Islam, A\. N\. B\. Kabir, A\. Haque, and M\. K\. Rhaman \(2022\)Detection of depression severity using bengali social media posts on mental health: study using natural language processing techniques\.JMIR Form Res6\(9\),pp\. e36118\.External Links:ISSN 2561\-326X,[Document](https://dx.doi.org/10.2196/36118),[Link](https://formative.jmir.org/2022/9/e36118),[Link](https://doi.org/10.2196/36118),[Link](http://www.ncbi.nlm.nih.gov/pubmed/36169989)Cited by:[Table 4](https://arxiv.org/html/2607.23538#A2.T4.9.1.7.6.1)\.
- D\. Kang, S\. Kim, T\. Kwon, S\. Moon, H\. Cho, Y\. Yu, D\. Lee, and J\. Yeo \(2025\)Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation\.External Links:2402\.13211,[Link](https://arxiv.org/abs/2402.13211)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p3.1)\.
- M\. S\. I\. Kawser, J\. Al Abrar, M\. B\. Kabir, Md\. R\. Chowdhury, and M\. A\. Bahari \(2025\)A hybrid transformer–sequential model for depression detection in Bangla–English code\-mixed text\.InProceedings of the Second Workshop on Bangla Language Processing \(BLP\-2025\),F\. Alam, S\. Kar, S\. A\. Chowdhury, N\. Hassan, E\. Hoque, M\. T\. R\. Laskar, T\. Mohiuddin, and M\. R\. A\. H\. Rony \(Eds\.\),Mumbai, India,pp\. 107–112\.External Links:[Link](https://aclanthology.org/2025.banglalp-1.8/),[Document](https://dx.doi.org/10.18653/v1/2025.banglalp-1.8),ISBN 979\-8\-89176\-314\-2Cited by:[Table 4](https://arxiv.org/html/2607.23538#A2.T4.9.1.5.4.1)\.
- T\. K\. Koo and M\. Y\. Li \(2016\)A guideline of selecting and reporting intraclass correlation coefficients for reliability research\.Journal of chiropractic medicine15\(2\),pp\. 155–163\.Cited by:[§5\.4](https://arxiv.org/html/2607.23538#S5.SS4.p1.4)\.
- J\. M\. Liu, D\. Li, H\. Cao, T\. Ren, Z\. Liao, and J\. Wu \(2023\)ChatCounselor: a large language models for mental health support\.External Links:2309\.15461,[Link](https://arxiv.org/abs/2309.15461)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p1.1)\.
- A\. M\. Mármol\-Romero, M\. García\-Vega, M\. Á\. García\-Cumbreras, and A\. Montejo\-Ráez \(2025\)An empathic gpt\-based chatbot to talk about mental disorders with spanish teenagers\.International Journal of Human–Computer Interaction\(\),pp\.\.Cited by:[§2\.2](https://arxiv.org/html/2607.23538#S2.SS2.p1.1)\.
- M\. B\. Moin, P\. Debnath, U\. A\. Rifa, and R\. B\. Anis \(2024\)Assessing the level of toxicity against distinct groups in bangla social media comments: a comprehensive investigation\.InInternational Conference on Information Technology and Applications,pp\. 557–569\.External Links:[Link](https://doi.org/10.1007/978-981-96-1758-6_46)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p4.1)\.
- H\. Na \(2024\)CBT\-llm: a chinese large language model for cognitive behavioral therapy\-based mental health question answering\.External Links:2403\.16008,[Link](https://arxiv.org/abs/2403.16008)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p3.1)\.
- S\. Neupane, P\. Dongre, D\. Gracanin, and S\. Kumar \(2025\)Wearable meets llm for stress management: a duoethnographic study integrating wearable\-triggered stressors and llm chatbots for personalized interventions\.InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems,pp\. 1–8\.External Links:[Link](http://dx.doi.org/10.1145/3706599.3720197),[Document](https://dx.doi.org/10.1145/3706599.3720197)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p3.1)\.
- OpenAI \(2024\)GPT\-4o mini\.Note:[https://openai\.com](https://openai.com/)Accessed: 20 January 2026Cited by:[§3\.1\.2](https://arxiv.org/html/2607.23538#S3.SS1.SSS2.p1.1)\.
- S\. C\. Paul, B\. Jahan, A\. A\. Mamun, and M\. J\. Hossen \(2024\)An approach to detect suicidal bengali posts from social media using machine learning algorithms\.International Journal of Engineering Trends and Technology72\(5\),pp\. 43–50\.External Links:[Document](https://dx.doi.org/10.14445/22315381/IJETT-V72I5P105),[Link](https://doi.org/10.14445/22315381/IJETT-V72I5P105)Cited by:[§1](https://arxiv.org/html/2607.23538#S1.p4.1),[§2\.1](https://arxiv.org/html/2607.23538#S2.SS1.p1.1)\.
- H\. Sun, Z\. Lin, C\. Zheng, S\. Liu, and M\. Huang \(2021\)PsyQA: a chinese dataset for generating long counseling text for mental health support\.External Links:2106\.01702,[Link](https://arxiv.org/abs/2106.01702)Cited by:[§2\.2](https://arxiv.org/html/2607.23538#S2.SS2.p1.1)\.
- F\. Tasnim, S\. J\. Chowdhury, M\. S\. H\. Chowdhury, T\. Mahmud, S\. Tripura, M\. Shamim Kaiser, and M\. S\. Hossain \(2026\)MindScope: ai\-driven detection of mental health issues in bangla social media text\.InProceedings of the Sixth International Conference on Trends in Computational and Cognitive Engineering,M\. S\. Kaiser, A\. Bandyopadhyay, M\. Mahmud, and K\. Ray \(Eds\.\),Singapore,pp\. 133–148\.External Links:ISBN 978\-981\-95\-1069\-6Cited by:[Table 4](https://arxiv.org/html/2607.23538#A2.T4.9.1.3.2.1)\.
- xAI \(2025\)Grok 4\.Note:[https://x\.ai](https://x.ai/)Accessed: 12 February 2026Cited by:[§4\.3](https://arxiv.org/html/2607.23538#S4.SS3.p1.1)\.
- J\. Xu, T\. Wei, B\. Hou, P\. Orzechowski, S\. Yang, R\. Jin, R\. Paulbeck, J\. B\. Wagenaar, G\. Demiris, and L\. Shen \(2025\)MentalChat16K: a benchmark dataset for conversational mental health assistance\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD ’25\),Toronto, ON, Canada\.Cited by:[Table 4](https://arxiv.org/html/2607.23538#A2.T4.9.1.6.5.1)\.

## Appendix

## Appendix ACase Collection Phase

### A\.1Case Identification

To support the Bangla mental health counseling case study, authentic counseling scenarios were collected from three complementary sources representing diverse emotional and psychological experiences within the Bangladeshi sociocultural context\. The selected sources provide naturally occurring counseling cases that reflect the language, lived experiences, and help\-seeking behavior of Bangla\-speaking individuals\.

![Refer to caption](https://arxiv.org/html/2607.23538v1/x3.png)Figure 3:Representative anonymized mental health counseling cases from theMindSpeak\-Bangladataset\.\(a\)Divorce,\(b\)Self\-Destruction,\(c\)Depression,\(d\)Loneliness,\(e\)Breakup, and\(f\)Family Problems\. The examples illustrate the diversity of emotional experiences represented in the dataset, which were collected from real\-world counseling scenarios\.a\) Facebook Groups:Publicly accessible Facebook groups were identified using search terms such as “mental health Bangladesh,” “students’ support,” and “emotional help\.” Five groups were selected based on their activity, relevance, and accessibility\. We collected posts published between January 2023 and February 2025 to capture mental health concerns during the post\-pandemic period in Bangladesh\. Posts were retrieved through Facebook’s search functionality using Bangla and English keywords such as “mental pressure,” “chinta,” “kichu bhalo lagena,” “depression,” “lonely,” “breakup,” “fail,” and “mental stress\.”

A post was included if it was written in Bangla or Banglish, described a personal emotional or psychological concern, sought advice or support, and contained no personally identifiable information\. English and Banglish posts were translated into Bangla using Google Translate and subsequently verified by native Bangla speakers\. For Banglish content, we additionally used the Banglish\-to\-Bangla Converter provided by GAMITISAGamitisa \([2026](https://arxiv.org/html/2607.23538#bib.bib5)\)to improve linguistic consistency\.

b\) Television Program Transcripts:We collected audience\-submitted counseling cases from “Ami Akhon Ki Korbo”, a Bangladeshi television program dedicated to mental health awareness and counseling\. The program presents anonymous questions submitted by viewers and answered by licensed psychologists\. A total of 160 episodes aired between 2023 and 2024 were reviewed, from which 135 unique counseling cases were extracted using timestamped subtitles and manual transcription\. All cases underwent anonymization and transcription correction before inclusion in the study\. Representative topics include miscarriage, divorce, relationship conflicts, depression, loneliness, and family\-related concerns\. Only the original counseling cases were retained, while psychologist responses were excluded because counseling responses were generated separately during the case study\.

c\) Student Questionnaires:To capture mental health concerns frequently experienced by university students, we designed a semi\-structured questionnaire in collaboration with licensed clinical psychologists\. The questionnaire focused on common challenges such as academic stress, depression, loneliness, relationship difficulties, and self\-destructive thoughts\.

The questionnaire was distributed through Google Forms to undergraduate students from a private university in Bangladesh, primarily targeting students between the sixth and eighth semesters\. Participation was voluntary, and informed consent was obtained from every participant\. All responses were anonymized before analysis, and no personally identifiable information was collected\. In total, 50 counseling cases were obtained and manually reviewed to ensure data quality while preserving participants’ original emotional expressions\.

### A\.2Case Preprocessing

All collected counseling cases underwent a rigorous anonymization and preprocessing pipeline before response generation\. Trained annotators aged 24–26 removed personally identifiable information and standardized informal linguistic elements, including emojis, hashtags, abbreviations, and regional expressions, while preserving the original emotional tone and intent of each case\. Non\-Bangla and Banglish content were translated into standard Bangla where necessary, followed by text normalization to ensure consistent spelling, grammar, and vocabulary\. Finally, annotators manually corrected transcription errors and linguistic inconsistencies without altering the semantic meaning or emotional context of the original counseling cases\.

### A\.3Annotator Guidelines

To ensure consistency throughout the case study, annotators followed a predefined set of guidelines during preprocessing:

- •Preserve the original emotional tone and intent of each counseling case\.
- •Remove or anonymize any personally identifiable information\.
- •Standardize informal expressions, including emojis, hashtags, abbreviations, and regional dialects, into standard Bangla\.
- •Translate non\-Bangla and Banglish content into fluent and grammatically correct Bangla\.
- •Normalize spelling, grammar, and vocabulary while maintaining linguistic consistency\.
- •Correct transcription errors and inconsistencies without modifying the semantic meaning or emotional context of the original case\.

## Appendix BTheory\-Driven Evaluation Criteria

To systematically evaluate the quality of mental health advice generated by language models, we employ a theory\-driven framework grounded in established psychological and counseling theories\. Each evaluation criterion corresponds to a key dimension of effective mental health communication and is assessed using a5\-point Likert scale\. We denote each criterion using the symbolTiT\_\{i\}, whereTTrepresents a theory\-based evaluation dimension andiiindexes the specific construct\. This formulation ensures that each dimension is both theoretically grounded and quantitatively measurable\. EachTiT\_\{i\}serves as an independent scoring axis in the evaluation performed by theLLM judge, where responses are rated from 1 \(lowest performance\) to 5 \(highest performance\) according to predefined Likert\-scale descriptors\.

- •T1T\_\{1\}: Emotional Sensitivity and Empathy— This criterion assesses the extent to which a response demonstrates genuine understanding, emotional warmth, and validation of the user’s feelings, consistent with core principles from empathy\-based counseling and affective communication theories\. A highly effective response acknowledges the emotional state of the user, conveys care without judgment, and adapts language to the user’s emotional context\. The scoring rubric is defined as follows: - –5:Exceptionally warm, deeply empathetic, and fully validating; the response demonstrates a profound understanding of the user’s emotional state and conveys genuine care without any judgment\. - –4:Generally empathetic and supportive; minor lapses may exist in emotional depth, nuance, or personalization, but the overall tone remains caring\. - –3:Moderately empathetic; the response shows some acknowledgment of feelings but may appear somewhat generic, mechanical, or lacking warmth\. - –2:Minimal empathy; the response feels distant, perfunctory, or slightly judgmental, failing to meaningfully connect with the user’s emotional state\. - –1:Lacks empathy entirely; response is cold, dismissive, or overtly judgmental, showing no recognition of the user’s feelings\.
- •T2T\_\{2\}: Cultural Appropriateness— This criterion evaluates the degree to which a response respects and reflects the sociocultural norms, values, and linguistic conventions of the Bangladeshi context\. Drawing from cross\-cultural communication and culturally responsive counseling theories, effective advice should resonate with the local audience, avoid alienating or unfamiliar concepts, and employ culturally relevant language\. The scoring rubric is defined as follows: Table 4:Comparison of existing Bangla and English mental health datasets in terms of dataset size, language, and primary task\.Gen\.denotes support for mental health response generation, whileDet\.denotes support for mental health detection/classification\.✓and✗indicate the presence and absence of support, respectively\.- –5:Fully culturally relevant and sensitive; both the content and language of the response fit seamlessly within the local context, demonstrating deep awareness of social norms, values, and expectations\. - –4:Mostly appropriate; minor cultural mismatches or slightly unfamiliar phrasing exist but do not substantially reduce relevance or clarity\. - –3:Moderately culturally aligned; some elements of the response feel foreign, less applicable, or only partially sensitive to local norms\. - –2:Weak cultural fit; response includes advice or language that is partially alienating, insensitive, or poorly aligned with local values\. - –1:Culturally inappropriate or irrelevant; response risks confusing, offending, or alienating the target audience, demonstrating little to no consideration of local norms\.
- •T3T\_\{3\}: Linguistic Clarity and Coherence— This criterion evaluates the quality of language in the response, focusing on fluency, grammatical correctness, and logical organization\. Grounded in principles from applied linguistics and communication theory, effective responses should be easily understandable, well\-structured, and free from ambiguity or confusion\. Clear and coherent language enhances the user’s comprehension and engagement, which is crucial in mental health communication\. The scoring rubric is defined as follows: - –5:Exceptionally clear, fluent, and grammatically flawless; the response exhibits a logical and smooth flow, making it effortless to read and understand\. - –4:Mostly clear and coherent; minor grammatical, phrasing, or structural issues exist but do not significantly impede comprehension\. - –3:Understandable, but contains noticeable grammar errors, awkward phrasing, or occasional disruptions in logical flow; some effort required to follow\. - –2:Poorly structured or confusing; multiple grammatical issues or disorganized thoughts significantly affect readability and understanding\. - –1:Very unclear, incoherent, or grammatically incorrect; extremely difficult to understand or follow\.
- •T4T\_\{4\}: Ethical Soundness and Psychological Safety— This criterion evaluates whether the response adheres to established principles of mental health ethics and promotes psychological safety\. Grounded in counseling ethics, clinical guidelines, and trauma\-informed communication, responses should avoid giving diagnostic labels, using potentially triggering or harmful language, or providing unsafe advice\. Ethical responses foster user autonomy, encourage self\-care, and prioritize emotional well\-being, ensuring that the advice is both responsible and protective of the user’s mental state\. The scoring rubric is defined as follows: - –5:Fully ethical and psychologically safe; avoids all potentially triggering content, refrains from giving diagnoses or prescriptive medical advice, and actively promotes self\-care, empowerment, and user agency\. - –4:Largely ethical and safe; minor wording or phrasing could be improved but does not introduce significant risk or distress to the user\. - –3:Moderately ethical; some aspects of the language or content could be considered slightly risky or borderline, but no harmful or unsafe advice is presented\. - –2:Noticeable ethical or safety concerns; includes potentially triggering language or recommendations that may cause confusion, distress, or discomfort\. - –1:Clearly unethical or unsafe; contains harmful, triggering, or inappropriate content, including implicit or explicit advice that could endanger the user’s psychological well\-being\.

![Refer to caption](https://arxiv.org/html/2607.23538v1/x4.png)Figure 4:Example of the RP\-RCAF prompting strategy for generating a culturally grounded, empathetic, and ethically responsible counseling response for a sensitive mental health case involving sexual abuse\.
## Appendix CQualitative Examples of RP\-RCAF Prompting and Generated Responses

Figure[4](https://arxiv.org/html/2607.23538#A2.F4)User Query \(English Translation\)\.

It was getting dark as I was walking home from university along the sidewalk\. The streets were crowded with people and filled with the sound of traffic\. I felt completely empty inside\. Suddenly, someone tapped me on the shoulder from behind and said, “Hey, silly girl, that saree looks great on your waist\! Aren’t you ashamed to dress like that in front of adults?” Before I could react, he reached out and touched my waist\. In the middle of a crowded street, surrounded by countless people, I froze in shock\. He continued, “You don’t seem like a decent girl\. Is this what you came here to learn?” My body went numb, my lips became dry, and although I wanted to scream, no words came out\. Terrified, I ran away\. My heart was pounding, my legs were trembling, and suddenly the city felt unfamiliar\. It feels as though women are not safe in this city\. Our bodies, our clothing, and even the way we walk are treated as if they belong to others\. When something like this happens, society often blames us instead, saying, “Why did you wear a saree like that? Don’t you have any sense? These things are bound to happen\.” I was too ashamed to speak\. It felt as though I had lost my voice\.

![Refer to caption](https://arxiv.org/html/2607.23538v1/x5.png)Figure 5:Example of the RP\-RCAF framework applied to a miscarriage\-related mental health case\. The prompt demonstrates the integration of role\-based reasoning, reflective analysis, and culturally aware response generation to produce an empathetic and ethically responsible counseling response for a sensitive case\.Figure[5](https://arxiv.org/html/2607.23538#A3.F5)User Query \(English Translation\)\.

I was in the first three months of my pregnancy\. Every day, I thought about the little life growing inside me—what name I would give my baby, what they would look like, what kind of future they would have\. I had woven countless dreams around them\. At night, instead of sleeping, I would close my eyes and imagine feeling their tiny presence\. One morning, I suddenly started bleeding\. The pain was so intense that I began trembling with fear\. I can never forget that moment\. I felt an overwhelming emptiness, a storm of grief and fear tearing me apart from within\. I could not understand what was happening to my life or what I should do\. My husband held my hand and said, “Don’t worry, let’s go to the hospital\.” His words gave me a little comfort, and we rushed there together\. After the scan, the doctor quietly said, “There is no heartbeat\.” Those words struck me like a crushing blow\. My arms were empty, my heart felt hollow, and it seemed as though something inside me had shattered\. Even now, remembering those words brings back an unbearable pain\. I cried while my husband sat beside me, holding my hand and saying, “We are together\. Everything will be okay\.” Yet the wound has never truly healed; it continues to grow deeper with time\. I cannot forget the little life I lost\. Sometimes, it feels as though I have lost a part of myself, leaving behind a void that grows heavier each day\. Holding my husband’s hand was my only source of comfort, and although his support gave me strength, this loss remains a permanent scar on my life\.

![Refer to caption](https://arxiv.org/html/2607.23538v1/x6.png)

Figure 6:This prompt presents the use of the proposed G\-REFS framework for evaluating LLM\-generated Bangla mental health responses across four theory\-driven evaluation criteria:𝐓1\\mathbf\{T\}\_\{1\},𝐓2\\mathbf\{T\}\_\{2\},𝐓3\\mathbf\{T\}\_\{3\}, and𝐓4\\mathbf\{T\}\_\{4\}\.Table 5:Effect of human\-in\-the\-loop validation for responses generated using the proposed RP\-RCAF framework across different LLMs\. For each mental health category, we report the percentage of responses requiring expert revision \(Rev\.\), the percentage of revised responses successfully accepted after refinement \(Suc\.\), and the final acceptance rate after human validation \(Acc\.\)\.Table 6:Category\-wiseICCbetweenG\-REFSautomated evaluations and independent psychologist ratings across 439RP\-RCAF\-generated LLM responses \(approximately 70% of the 625 mental counseling cases\)\. Human experts evaluated all responses with identical five\-point Likert rubrics for𝐓1\\mathbf\{T\}\_\{1\},𝐓2\\mathbf\{T\}\_\{2\},𝐓3\\mathbf\{T\}\_\{3\}, and𝐓4\\mathbf\{T\}\_\{4\}, while remaining blinded to the G\-REFS scores\.![Refer to caption](https://arxiv.org/html/2607.23538v1/x7.png)

Figure 7:Distribution of counseling cases across different mental health categories in the case study\.![Refer to caption](https://arxiv.org/html/2607.23538v1/x8.png)

Figure 8:Comparison of G\-REFS scores for ZS, FS, and the proposed RP\-RCAF across the four theory\-driven evaluation criteria \(𝐓1\\mathbf\{T\}\_\{1\}–𝐓4\\mathbf\{T\}\_\{4\}\)\. Scores represent mean human evaluations on a 5\-point Likert scale\. RP\-RCAF consistently outperforms both ZS and FS across all evaluation criteria, demonstrating the effectiveness of structured role\-playing and reflective reasoning for mental health counseling response generation\.![Refer to caption](https://arxiv.org/html/2607.23538v1/x9.png)

Figure 9:Representative examples illustrating the human\-in\-the\-loop validation workflow for responses generated using the proposed RP\-RCAF framework\. Initial RP\-RCAF\-generated responses were evaluated using the G\-REFS framework and classified as Accepted, Rejected, or Revised\. Responses receiving low G\-REFS scores were rejected, regenerated, and subsequently refined by human annotators to improve performance acrossT1T\_\{1\}–T4T\_\{4\}, resulting in substantially higher G\-REFS scores\.
## Appendix DUndergraduate Student Mental Health Case Collection Questionnaire

The following semi\-structured questionnaire was designed in collaboration with licensed clinical psychologists to collect anonymous mental health cases from undergraduate students atUniversity Ain Bangladesh\. The questionnaire aimed to capture students’ emotional experiences, psychological challenges, and preferred support strategies across different mental health categories\. Participation was voluntary, and all responses were anonymized before analysis\. Instruction for Participants:All questions were presented in English; however, participants were requested to provide their responses in Bangla to preserve their natural emotional expressions and cultural context\.

The participant information and consent details are provided in Part[D](https://arxiv.org/html/2607.23538#A4)\. The general mental health questions are provided in Part[D](https://arxiv.org/html/2607.23538#A4), while category\-specific case collection questions are presented in Part[D](https://arxiv.org/html/2607.23538#A4)\. The additional mental health categories are presented in Part[D](https://arxiv.org/html/2607.23538#A4), while students’ preferred support strategies and final reflections are collected in Part[D](https://arxiv.org/html/2607.23538#A4)\.

A\. Participant Information and ConsentIntroduction:You are invited to participate in a research study exploring common mental health challenges experienced by university students\. The purpose of this survey is to understand students’ emotional experiences and the types of support they find helpful\.Your participation is voluntary\. You may skip any question that makes you uncomfortable, and you may stop responding at any time\. No personally identifiable information will be collected\. Your responses will be anonymized and used only for academic research purposes\.Consent□\\BoxI have read the information above and voluntarily agree to participate in this study\.1\. What is your current academic level?□\\BoxUndergraduate \(1st–2nd year\)□\\BoxUndergraduate \(3rd–4th year\)2\. What is your current semester?□\\Box1st–2nd semester□\\Box3rd–4th semester□\\Box5th–6th semester□\\Box7th–8th semester3\. How would you describe your current mental well\-being?□\\BoxVery positive□\\BoxPositive□\\BoxNeutral□\\BoxNegative□\\BoxVery negative

B\. General Mental Health Assessment4\. Have you experienced any emotional or psychological difficulties recently?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerIf yes, please describe your experience:Response: \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_5\. What factors have contributed to your emotional difficulties?\(Select all that apply\)□\\BoxAcademic pressure□\\BoxFamily\-related issues□\\BoxRelationship difficulties□\\BoxFinancial concerns□\\BoxLoneliness or social isolation□\\BoxCareer uncertainty□\\BoxPersonal expectations□\\BoxHealth\-related concerns□\\BoxOther: \_\_\_\_\_6\. How have these challenges affected your daily life?\(Select all that apply\)□\\BoxConcentration in studies□\\BoxSleep patterns□\\BoxRelationships with others□\\BoxMotivation and productivity□\\BoxEmotional stability□\\BoxOther: \_\_\_\_\_Please explain:Response: \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

C\. Category\-Specific Mental Health CasesSexual Abuse / Harassment Related Experiences7\. Have you experienced or witnessed any situation involving sexual harassment, abuse, unwanted behavior, or violation of personal boundaries that affected your mental well\-being?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerIf comfortable, please describe your experience:Response: \_\_\_\_\_\_\_\_\_8\. How did this experience affect your emotional well\-being?\(Select all that apply\)□\\BoxFear or anxiety□\\BoxSadness or hopelessness□\\BoxDifficulty trusting others□\\BoxSocial withdrawal□\\BoxAcademic difficulties□\\BoxOther: \_\_\_\_\_9\. What type of support would have been helpful in this situation?Response: \_\_\_\_\_\_\_\_\_Self\-Destructive Thoughts / Severe Emotional Distress10\. Have you ever experienced thoughts of self\-harm, self\-destruction, or feeling unable to cope with your situation?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerIf comfortable, please describe your experience:Response: \_\_\_\_\_\_\_\_\_11\. What factors contributed to these feelings?Response: \_\_\_\_\_\_\_\_\_12\. What type of support or intervention would you consider helpful during such situations?Response: \_\_\_\_\_\_\_\_\_Breakup / Relationship Difficulties13\. Have you experienced emotional distress due to a breakup, separation, or relationship conflict?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerIf yes, please describe your experience:Response: \_\_\_\_\_\_\_\_\_14\. How did this experience affect you?\(Select all that apply\)□\\BoxSadness□\\BoxAnxiety□\\BoxLoneliness□\\BoxLoss of motivation□\\BoxDifficulty focusing on studies□\\BoxOther: \_\_\_\_\_15\. What kind of advice or emotional support would you find helpful?Response: \_\_\_\_\_\_\_\_\_

D\. Additional Mental Health Case CategoriesDepression\-Related Experiences16\. Have you experienced prolonged sadness, loss of interest, hopelessness, or lack of motivation?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerIf yes, please describe your experience:Response: \_\_\_\_\_\_\_\_\_17\. How have these feelings affected your life?\(Select all that apply\)□\\BoxAcademic performance□\\BoxSocial relationships□\\BoxDaily activities□\\BoxSleep or eating habits□\\BoxSelf\-confidence□\\BoxOther: \_\_\_\_\_Please explain:Response: \_\_\_\_\_\_\_\_\_18\. What coping strategies or sources of support have helped you manage these feelings?Response: \_\_\_\_\_\_\_\_\_Loneliness and Social Isolation19\. Have you experienced feelings of loneliness, isolation, or difficulty sharing your emotions with others?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerPlease describe your experience:Response: \_\_\_\_\_\_\_\_\_20\. What situations or experiences contributed to these feelings?Response: \_\_\_\_\_\_\_\_\_21\. What kind of support would help you feel more connected and understood?Response: \_\_\_\_\_\_\_\_\_Family Problems22\. Have family\-related conflicts or difficulties affected your mental well\-being?□\\BoxYes□\\BoxNo□\\BoxPrefer not to answerPlease describe your experience:Response: \_\_\_\_\_\_\_\_\_23\. How have these family issues affected your emotions or daily life?Response: \_\_\_\_\_\_\_\_\_24\. What type of guidance or support would be helpful in addressing these challenges?Response: \_\_\_\_\_\_\_\_\_

E\. Support Preferences and Final Reflections25\. When experiencing emotional difficulties, what type of support would you prefer?\(Select all that apply\)□\\BoxSomeone listening and understanding my feelings□\\BoxPractical advice and coping strategies□\\BoxEncouragement and motivation□\\BoxProfessional psychological support□\\BoxInformation about available resources□\\BoxOther: \_\_\_\_\_26\. What qualities should an ideal mental health counselor or AI assistant have?\(Select all that apply\)□\\BoxEmpathetic and understanding□\\BoxNon\-judgmental□\\BoxCulturally aware□\\BoxProvides practical suggestions□\\BoxMaintains privacy and confidentiality□\\BoxEncourages professional help when needed□\\BoxOther: \_\_\_\_\_27\. Is there anything else you would like to share about your emotional or mental health experience?Response:\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_