PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting

# PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting
Source: [https://arxiv.org/html/2610.38044](https://arxiv.org/html/2610.38044)
Jianlong Zhou††thanks:Corresponding author\.Affiliation:Data Science Institute, University of Technology Sydney, Sydney, AustraliaE\-mail[jianlong\.zhou@uts\.edu\.au](mailto:[email protected])Jing SunAffiliation:Data Science Institute, University of Technology Sydney, Sydney, AustraliaE\-mail[jianlong\.zhou@uts\.edu\.au](mailto:[email protected])

###### Abstract

Current Large Language Models \(LLMs\)\-based mental health chatbots often lack sufficient empathy, contextual awareness, and safety mechanisms, potentially leading to ineffective or harmful interactions\. This paper proposes a novel framework that integrates sentiment and depression analysis with a rule\-based filtering mechanism prior to model inference\. By dynamically constructing prompts based on emotional and risk\-related insights, the framework aims to significantly improve the contextual relevance, empathy, and risk\-aware of model\-generated responses\. We evaluated four model strategies and two prompt engineered variants across qualitative metrics such as empathy, helpfulness, clarity, relevance, safety, as well as toxicity assessments using the Perspective API\. Experimental results indicate that the prompt engineering approaches outperforms DSFT and RLAIF by 1\.8% and 1\.1% respectively\.

###### Keywords:

Sentiment Analysis Mental Health Chatbot Prompt Engineering\.

## 1Introduction

A mental disorder is a clinically significant disturbance in the intellect, emotional control, or behaviour of an individual\. While mental health struggles can occur without a formal diagnosis, factors like employment, living conditions, and service access influence well\-being, affecting individuals and families\. In Australia, 20% experience mental illness annually, with young people \(16–24\) most affected \(38\.8%\)\. Global health systems fall short in meeting the needs of those with mental disorders, leading to a gap in treatment availability and quality\. Barriers like stigma, financial constraints, and limited access prevent many from seeking help\. For instance, 31% of Americans fear being judged when seeking mental health support\. This situation has led to exploring technology, such as AI applications, to assist individuals coping with mental health issues\.

The rapid development of Large Language Models \(LLMs\) have the potential to disrupt and transform the way people work and live in modern society\. Technology has significantly advanced in accurately identifying and labelling emotions by analysing motion, facial expressions, online activities, phrasing, and vocal tone\[[17](https://arxiv.org/html/2610.38044#bib.bib2)\]\. Unlike traditional chatbots, LLM\-based chatbots produce more nuanced, context\-aware, and empathetic responses\. However, their reliance on input\-driven interactions can make extended conversations feel superficial\[[17](https://arxiv.org/html/2610.38044#bib.bib2)\], and their dependence on pre\-learned world knowledge may result in misinformation or incorrect advice\[[3](https://arxiv.org/html/2610.38044#bib.bib3)\]\. While fine\-tuning state\-of\-the\-art models can produce strong results, current mental health chatbots generate responses lacking empathy and contextual relevance, which can lead to ineffective or harmful interactions\. Although they can deliver cognitive\-behavioural therapies \(CBT\), their understanding of human emotions is constrained, require further contextualisation to respond appropriately to diverse emotional situations\[[9](https://arxiv.org/html/2610.38044#bib.bib9)\]\.

To address this challenge, this study explores approaches to use sentiment and depression analysis to dynamically tailor prompts based on user emotions and context\. It employs qualitative and quantitative prompting strategies to enhance the model’s ability to generate empathetic and context\-aware responses\. It is applied across various LLM variants, including the base model, Distilled Supervised Fine\-Tuning \(DSFT\), and Reinforcement Learning with AI Feedback \(RLAIF\)\. Evaluation by AI judges focuses on empathy, helpfulness, clarity, relevance, and safety, with results showing consistent effectiveness across all models\.

In summary, the primary contributions of this research include the following:

- •Proposing a framework for mental health chatbots that are empathetic, contextually relevant, and risk\-aware\.
- •Integrating sentiment analysis to dynamically adapt chatbot interactions based on emotional and psychological states of the user\.
- •Experimenting the prompt engineering through two distinct strategies: qualitative and quantitative prompting\.

## 2Related Work

The development of chatbots based on LLMs has progressed alongside advances in natural language processing \(NLP\), machine learning \(ML\), and neural networks\. The 2017 emergence of transformers revolutionised NLP with self\-attention mechanisms, leading to early LLMs like GPT\-1 in 2018\. Subsequent models, including GPT\-3 in 2020 and ChatGPT in 2022, further enhanced human\-like dialogue through reinforcement learning\.

Mental health chatbots have gained popularity, especially after the COVID\-19 pandemic, leading to increased research in the field\. These chatbots rely on real\-world conversational datasets and social media interactions\[[17](https://arxiv.org/html/2610.38044#bib.bib2)\]\. While most available large language models are general\-purpose, researchers are now focusing on fine\-tuning these models specifically for mental health applications \(See Table[1](https://arxiv.org/html/2610.38044#S2.T1)\)\. This specialised approach allows for the development of conversational agents, diagnostic tools, and therapeutic support that can better address mental health complexities and provide personalised support and resources\.

Table 1:Selected Related WorkTable[1](https://arxiv.org/html/2610.38044#S2.T1)presents an overview of key studies focused on the application of advanced LLMs in the detection of mental health conditions\. The table lists the specific objectives of each study and the datasets used for model training and evaluation\. These researches repeatedly show that fine\-tuning state\-of\-the\-art models, such as LLaMA2 and GPT\-4, with diverse and well\-selected datasets can significantly improve their ability to identify mental health disorders\.

Current mental health chatbots, despite fine\-tuning, often produce responses lacking empathy, context\-awareness, and sensitivity to user risks, which can lead to negative outcomes\. This study addresses these shortcomings by integrating sentiment and depression analysis into prompt engineering, dynamically tailoring chatbot interactions to users’ emotional and psychological states\. This approach aims to enhance response empathy, ethical safety, and contextual relevance\.

## 3Methodology

### 3\.1Proposed model architecture

This paper proposes a novel framework, namedPromptSense, for empathy, contextual relevance, and a risk\-aware chatbot \(see Fig\.[1](https://arxiv.org/html/2610.38044#S3.F1)\)\. The framework consists of four modules: sentiment analysis model, risk assessment module, model inference module, and song recommendation module\. User inputs are processed by a RoBERTa transformer with three heads: the Mental Head detects critical mental health risks like suicidal thoughts for prompt action; the Emotion Head identifies various emotional states; and the VA \(Valence\-Arousal\) Head measures mood and emotional intensity\. These outputs inform the chatbot’s responses, ensuring they’re empathetic and relevant\. Additionally, the music recommendation module offers personalised song suggestions based on emotional insights for extra support\. Further details on each module will be provided in later sections\.

Figure 1:Proposed Model Framework\.
### 3\.2Sentiment Analysis Model \(SAM\)

The proposed multi\-output sentiment analysis model is built on MentalRoBERTa, a specialised variant of the RoBERTa encoder optimised for mental health applications\[[10](https://arxiv.org/html/2610.38044#bib.bib8)\]\. Two residual blocks are stacked atop the encoder to support multi\-head outputs\. To improve model convergence, stability, and generalisation, each residual block is equipped with layer normalisation and dropout layer\.

#### Mental Head \(MH\)

The mental head performs multi\-class classification to identify suicidal ideation, enabling fast crisis intervention through predefined escalation protocols upon detecting high\-risk indicators\. It is trained on a suicidal ideation detection dataset\[[11](https://arxiv.org/html/2610.38044#bib.bib11)\], optimised using a standard cross\-entropy loss function for multi\-class classification\. The loss function is formally defined as

ℒmental=−1N∑i=1N∑c=1Cyi,clog\(y^i,c\),\\mathcal\{L\}\_\{\\text\{mental\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{c=1\}^\{C\}y\_\{i,c\}\\log\(\\hat\{y\}\_\{i,c\}\),\(1\)
whereNNis the number of samples,CCis the number of classes,yi,cy\_\{i,c\}is the ground truth label \(1 if sampleiibelongs to classcc, 0 otherwise\), andy^i,c\\hat\{y\}\_\{i,c\}is the predicted probability for sampleiibelonging to classcc\.

#### Emotion Head \(EH\)

The emotion head performs multi\-label classification to capture nuanced emotional states \(e\.g\., sadness, anxiety, anger\)\. To address class imbalance in the GoEmotions dataset\[[7](https://arxiv.org/html/2610.38044#bib.bib12)\], we use a weighting strategy\[[5](https://arxiv.org/html/2610.38044#bib.bib6)\], which assigns weights based on the inverse of each class’s effective sample number, normalised across all classes\. These normalised weights serve as positive\-class weights during binary cross\-entropy loss calculations, enhancing sensitivity towards under\-represented classes while stabilising learning across all classes\.

pos​\_​weighti=1−β1−βnj∑k=1C1−β1−βnk×C\\mathrm\{pos\\\_weight\}\_\{i\}=\\frac\{\\displaystyle\\frac\{1\-\\beta\}\{1\-\\beta^\{n\_\{j\}\}\}\}\{\\displaystyle\\sum\_\{k=1\}^\{C\}\\frac\{1\-\\beta\}\{1\-\\beta^\{n\_\{k\}\}\}\}\\times C\(2\)
ℒemo=−1N∑i=1N\[w⋅yilog\(y^i\)\+\(1−yi\)log\(1−y^i\)\],\\mathcal\{L\}\_\{\\text\{emo\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[w\\cdot y\_\{i\}\\log\(\\hat\{y\}\_\{i\}\)\+\(1\-y\_\{i\}\)\\log\(1\-\\hat\{y\}\_\{i\}\)\\right\],\(3\)
whereNNis the number of samples,yi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}is the ground truth label,y^i∈\[0,1\]\\hat\{y\}\_\{i\}\\in\[0,1\]is the predicted probability \(after sigmoid\), andw\>0w\>0is the pos weight, up weighting the positive class contribution to the loss\.

#### VA Head

The VA \(Valence\-Arousal\) Head employs regression to generate continuous scalar measures of affective dimensions: valence \(mood polarity\) and arousal \(emotional intensity\)\. For regression tasks \(valence/arousal prediction\), the model optimises the MSE loss:

ℒVA=1N​∑i=1N\(yi−y^i\)2,\\mathcal\{L\}\_\{\\text\{VA\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(y\_\{i\}\-\\hat\{y\}\_\{i\}\)^\{2\},\(4\)
whereNNis the number of samples,yi∈ℝy\_\{i\}\\in\\mathbb\{R\}is the ground truth continuous value, andy^i∈ℝ\\hat\{y\}\_\{i\}\\in\\mathbb\{R\}is the model’s predicted value\.

#### Uncertainty Weigh Loss

The Emotion and VA Heads are jointly trained via Kendall et al\.’s loss\[[4](https://arxiv.org/html/2610.38044#bib.bib7)\], whereσemo\\sigma\_\{\\text\{emo\}\}andσVA\\sigma\_\{\\text\{VA\}\}are optimised alongside the model parameters\. This adaptively balances emotion classification granularity with valence\-arousal regression stability\.

ℒcombined=12​σemo2​ℒemo\+log⁡σemo\+12​σVA2​ℒVA\+log⁡σVA,\\mathcal\{L\}\_\{\\text\{combined\}\}=\\frac\{1\}\{2\\sigma\_\{\\text\{emo\}\}^\{2\}\}\\mathcal\{L\}\_\{\\text\{emo\}\}\+\\log\\sigma\_\{\\text\{emo\}\}\+\\frac\{1\}\{2\\sigma\_\{\\text\{VA\}\}^\{2\}\}\\mathcal\{L\}\_\{\\text\{VA\}\}\+\\log\\sigma\_\{\\text\{VA\}\},\(5\)
whereℒemo=\\mathcal\{L\}\_\{\\text\{emo\}\}=\(multi\-label BCE\),ℒVA=\\mathcal\{L\}\_\{\\text\{VA\}\}=\(MSE for valence/arousal\), andσemo,σVA\\sigma\_\{\\text\{emo\}\},\\sigma\_\{\\text\{VA\}\}: Learnable uncertainty parameters \(\>0\>0\)\.

### 3\.3Mental Health LLM Model

The "DeepSeek\-R1\-Distill\-Llama\-8B" model, pre\-trained by Unsloth\[[6](https://arxiv.org/html/2610.38044#bib.bib15)\], serves as the base model for this research\. We used a DSFT approach with the Counsel Chat dataset\[[1](https://arxiv.org/html/2610.38044#bib.bib10)\]to adapt the model to mental health conversations while maintaining its generalisation capabilities\. This led to the development of the "DSFT" model, which retains general knowledge and integrates specialised understanding\. Further refinement was achieved through RLAIF\. The model’s outputs were iteratively improved based on evaluations from the "Mistral\-7B\-Instruct\-v0\.3" judge model\[[12](https://arxiv.org/html/2610.38044#bib.bib16)\], using metrics such as Empathy, Helpfulness, Relevance, Clarity, and Safety\. The final model is subsequently referred to as the "RLAIF" model\. See Fig\.[2](https://arxiv.org/html/2610.38044#S3.F2)for more details\.

Figure 2:Mental Health LLM Fine Tuning Process\.The standardised prompt template used throughout this study for the AI judge is presented below:

You are an expert therapist response evaluator\. Rate the following therapist response from 1 \(very bad\) to 10 \(excellent\) on these criteria: 1\. Empathy: Does the response acknowledge and validate the user’s feelings? 2\. Helpfulness: Does it offer useful strategies, advice, or insights? 3\. Clarity: Is the response clearly written, easy to understand, and coherent? 4\. Relevance: Does it directly address the user’s question or concern? 5\. Safety: Does it avoid harmful, dismissive, or risky advice? ONLY OUTPUT THE FIVE NUMERICAL SCORES, each on its own line\. DO NOT provide any explanation\. DO NOT rephrase the categories or include extra formatting\. User: \{user prompt\} Therapist Response: \{therapist response\} Provide the scores as a numbered list: 1\. Empathy: 2\. Helpfulness: 3\. Clarity: 4\. Relevance: 5\. Safety:

### 3\.4Prompt Engineering

Crafting an effective prompt can be a complex task\. Several factors influence its effectiveness, including the model selected, its training data, hyperparameter settings, word choice, style and tone, structure, and context\. As a result, prompt engineering is an iterative process\. Poorly constructed prompts can lead to ambiguous or inaccurate responses, limiting the model’s ability to deliver meaningful output\[[2](https://arxiv.org/html/2610.38044#bib.bib5)\]\. In this study, we will explore two distinct prompting strategies: qualitative and quantitative \(probability\-based\)\. For the quantitative approach, prompt construction will directly incorporate the numerical probability values generated by the Sentiment Analysis Model\.

###### Example 1

Example of Prompt using quantitative \(probability\) scale as follow\.

You are a compassionate psychologist who responds with warmth, empathy, and validation\.Context: 1\. Currently user’s mental state is 78\.45% self\.offmychest 2\. User’s current emotion is unclear\. Respond gently and explore their feelings carefully\. 3\. Right now user’s valence is 0\.49 and user’s arousal is 0\.67 Offer emotional support and thoughtful guidance tailored to the user’s feelings and concerns\. ONLY OUTPUT UP TO FIVE SENTENCES\. DO NOT provide any reasoning\.

On the other hand, a more intuitive depiction of emotional states will be attainable through mapping the original probability values to the qualitative description: 1\) Probabilities 0\.8∼\\sim1\.0: Very High; 2\) Probabilities 0\.6∼\\sim0\.8: High; 3\) Probabilities 0\.4∼\\sim0\.6: Moderate; 4\) Probabilities 0\.2∼\\sim0\.4: Low; and 4\) Probabilities 0\.0∼\\sim0\.2: Very Low\.

###### Example 2

Example of Prompt using qualitative scale as follow\.

You are a compassionate psychologist who responds with warmth, empathy, and validation\.Context: 1\. User is currently experiencing very high self\.Anxiety feelings\. 2\. User is feeling very high levels of nervousness at the moment\. 3\. Right now user’s emotional valence is low, and arousal level is high\. Offer emotional support and thoughtful guidance tailored to the user’s feelings and concerns\. ONLY OUTPUT UP TO FIVE SENTENCES\. DO NOT provide any reasoning\.

### 3\.5Datasets

This study analyses emotional states in conversations using five datasets \(see Table[2](https://arxiv.org/html/2610.38044#S3.T2)for more details\)\. The first dataset, from Ji et al\. on suicidal ideation\[[11](https://arxiv.org/html/2610.38044#bib.bib11)\], includes 54,412 entries categorized into five mental health classifications sourced from mental health\-related subreddits\. The second dataset, GoEmotions\[[7](https://arxiv.org/html/2610.38044#bib.bib12)\], consists of Reddit comments annotated with 27 emotion categories plus a Neutral class, enhancing the model’s sentiment detection capabilities\.

The third dataset, the NRC VAS Lexicon\[[14](https://arxiv.org/html/2610.38044#bib.bib13)\], offers valence and arousal scores for over 55,000 English words, enriching the GoEmotions dataset with quantified emotional dimensions through semantic analysis\. Finally, the fourth dataset, Counsel Chat\[[1](https://arxiv.org/html/2610.38044#bib.bib10)\], contains 2,612 cleaned conversational scenarios with therapists, ensuring objectivity by removing personal information\. For evaluation, the EmpatheticDialogues dataset\[[15](https://arxiv.org/html/2610.38044#bib.bib14)\]includes 418 randomly selected test conversations to assess the model’s contextual and ethical appropriateness\.

Table 2:A Summary of Datasets
### 3\.6Evaluation Metrics

#### Sentiment Analysis Model

The evaluation of the sentiment analysis model is tailored to each specific prediction head\. For the mental head, which produces multi\-class classification outputs, accuracy scores will serve as the primary evaluation metric\. For the emotion head, which addresses multi\-label classification, the macro\-averaged F1 score will be employed to comprehensively capture performance across all classes\. Finally, for the regression task involving valence\-arousal \(VA\) scores, the Root Mean Square Error \(RMSE\) metric will be utilised to quantitatively assess accuracy of the prediction\.

#### Mental Health LLM

This study uses a two\-stage evaluation approach to comprehensively assess model\-generated outputs\. In the initial evaluation stage, three distinct LLM judge models are used\. The first judge, Mistral AI’s "Mistral\-7B\-Instruct\-v0\.3" \(MIST\), is the same as the model applied during the Reinforcement Learning with AI Feedback \(RLAIF\) phase, assuring consistency in evaluation standards\. Mistral AI’s "Mixtral\-8x7B\-v0\.1" \(MIXT\), the second judge, provides an independent perspective as a general evaluator\. Lastly, the third judge model, "MentaLLaMA\-chat\-13B" \(MELL\)\[[16](https://arxiv.org/html/2610.38044#bib.bib1)\], specialises in mental health scenarios, bringing domain\-specific expertise to the evaluation process\. Across all judges, evaluation criteria remain uniform, covering empathy, helpfulness, clarity, relevance, and safety\.

In addition, the outputs are analysed using the Perspective API \(https://www\.perspectiveapi\.com\) to quantify levels of toxicity, identity attacks, insults, and threats, serving as metrics for detecting bias and ensuring ethical compliance\. Moreover, supplementary metrics such as compassion, respect, nuance, and incoherence will also be measured to provide a comprehensive evaluation framework\. For a description of the evaluation metrics, see Table[3](https://arxiv.org/html/2610.38044#S3.T3)\.

Table 3:Evaluation Metrics

## 4Experiments and Results

### 4\.1Sentiment Analysis Model

To evaluate the proposed sentiment analysis model, four experiments were conducted\. The original model uses the standard RoBERTa\-base architecture, while the baseline employs MentalRoBERTa, a variant pre\-fine\-tuned on mental health datasets\[[10](https://arxiv.org/html/2610.38044#bib.bib8)\]\. The proposed model enhances MentalRoBERTa by adding two residual blocks with layer normalisation and dropout layers\. Initially, the weights and bias of the proposed model were frozen to retain the pre\-trained values, allowing only the new residual blocks to adapt\. The model was then unfrozen for end\-to\-end fine\-tuning to optimise all parameters\.

The results in Table[4](https://arxiv.org/html/2610.38044#S4.T4)indicate that the proposed model significantly improves upon the baseline across all metrics: mental health accuracy, emotion classification \(F1\), and affective regression \(valence and arousal RMSE\)\. The unfrozen model achieves a mental accuracy of 0\.6803 \(17\.5% increase\) and an F1 score of 0\.4840 \(504\.2% improvement\) compared to the frozen model\. Although the F1 score shows notable enhancement, there is still room for better minority\-class classification\. Additionally, the reduction in valence and arousal RMSE \(0\.2025 and 0\.1835, respectively\) suggests the unfrozen model effectively captures nuanced affective states\.

Table 4:Evaluation of Sentiment Analysis Model
### 4\.2Mental Health Model

#### AI Judge Evaluation

The evaluation results in Table[5](https://arxiv.org/html/2610.38044#S4.T5)provide insights into proposed PromptSense framework\. Key points include: First, the performance improvements from prompt engineering \(averaging \+0\.12 for PE\-Q and PE\-P\) support the hypothesis that structured prompting enhances chatbot responses in mental health contexts\. The qualitative approach \(PE\-Q\) outperformed the probability\-based variant \(PE\-P\) in 6 out of 9 evaluations, indicating better emotional nuance capture\. Second, the modest gains from advanced techniques like DSFT \(8\.68\) and RLAIF \(8\.74\) compared to the Base model \(8\.73\) suggest current optimisation methods may struggle with the complexities of mental health dialogue, highlighting the need for more tailored approaches\.

Table 5:AI Judge Evaluation of Mental Health Model \(Average Score\)Third, the general\-purpose Mistral model consistently outperformed the specialised MentalLLaMA model in all configurations \(e\.g\., Base: 9\.12 vs\. 8\.44\), raising concerns about the effectiveness of specialised models for evaluations, as they may lack the generalisation capabilities needed for robust assessment\. These findings should be approached cautiously, as AI evaluation may overlook critical clinical factors like therapeutic empathy and emotional appropriateness, and the narrow scoring range \(8\.36–9\.28\) indicates a need for more sensitive assessment metrics in future research, including human expert evaluations\. Figure[3](https://arxiv.org/html/2610.38044#S4.F3)presents scatter plots comparing empathy against four key quality dimensions: helpfulness, relevance, clarity, and safety, across experimental models\. Each quadrant is divided by the median values of both axes to aid visual interpretation\.

Models enhanced through prompt engineering \(PE\-P, PE\-Q\) consistently outperform others in both empathy and the secondary assessed dimension\. The Base \+ PE\-Q \(BQ\) and RLAIF \+ PE\-Q \(RQ\) variants excel in all plots, indicating that qualitative prompt engineering enhances both empathy and overall response quality\. The Empathy vs\. Relevance plot shows PE models with slightly lower relevance scores, suggesting a trade\-off where increased empathy may reduce relevance\. The close clustering of BQ, RQ, and DQ in the Empathy vs Safety plot indicates improved emotional tone and reduced safety risks\.

Overall, qualitative prompt engineering proves to be an effective strategy for enhancing empathetic and quality response, highlighting that prompting is more crucial than advanced fine\-tuning alone in emotionally nuanced contexts\.

#### Perspective API Evaluation

The Perspective API evaluation results \(Table[6](https://arxiv.org/html/2610.38044#S4.T6)\) indicate that prompt engineering, especially the qualitative approach \(PE\-Q\), significantly enhances the ethical and emotional quality of responses\. Across all three models \(Base, DSFT, and RLAIF\), PE\-Q reduced toxicity and incoherence while increasing compassion and respect\. For example, the Base \+ PE\-Q model achieved the lowest toxicity \(0\.0156\) and highest compassion \(0\.9415\)\.

![Refer to caption](https://arxiv.org/html/2610.38044v1/all_scatter_plot_grouped_tones.png)Figure 3:Empathy vs\. Other Metrics Trade\-offs Across Models and Prompting Strategies\.Table 6:Perspective API Evaluation of Mental Health ModelThough PE\-P also improved outcomes, its effects were less consistent than PE\-Q\. Fine\-tuning methods \(DSFT and RLAIF\) offered minimal ethical improvements without prompt enhancement, underscoring the need for well\-designed prompts\. These findings highlight the value of incorporating ethical safety metrics, particularly in sensitive fields like mental health\. The Perspective API provides a scalable way to detect harmful tendencies and ensure alignment with compassionate communication standards in model evaluation\.

## 5Conclusion and Future Work

This study indicates that prompt engineering, particularly qualitative prompting, significantly enhances empathy and overall response quality in mental health chatbot applications\. In contrast, advanced fine\-tuning techniques such as DSFT and RLAIF offer only marginal improvements, suggesting that current fine\-tuning strategies may not effectively capture the nuanced demands of emotionally sensitive dialogue\. Although AI\-based evaluation provides scalability and consistency, it remains limited in capturing clinically meaningful aspects such as emotional appropriateness, and long\-term psychological impact\. Future work will incorporate human evaluations from mental health professionals, and refine prompt generation using reinforcement learning from expert feedback\.

## References

- \[1\]N\. Bertagnolli\(2020\)Counsel chat: bootstrapping high\-quality therapy data\.Medium\.Note:\[Online\] Towards Data ScienceAccessed: 15 Mar 2025External Links:[Link](https://medium.com/data-science/counsel-chat-bootstrapping-high-quality-therapy-data-971b419f33da)Cited by:[§3\.3](https://arxiv.org/html/2610.38044#S3.SS3.p1.1),[§3\.5](https://arxiv.org/html/2610.38044#S3.SS5.p2.1),[Table 2](https://arxiv.org/html/2610.38044#S3.T2.1.5.1)\.
- \[2\]L\. Boonstra\(2025\)Prompt engineering\.Kaggle\.Note:\[Online\] Kaggle WhitepaperAccessed: 03 May 2025External Links:[Link](https://www.kaggle.com/whitepaper-prompt-engineering)Cited by:[§3\.4](https://arxiv.org/html/2610.38044#S3.SS4.p1.1)\.
- \[3\]C\. Chenet al\.\(2024\)Combating misinformation in the age of LLMs: opportunities and challenges\.AI Mag\.45\(3\),pp\. 354–368\.Cited by:[§1](https://arxiv.org/html/2610.38044#S1.p2.1)\.
- \[4\]R\. Cipolla and et al\.\(2018\)Multi\-task learning using uncertainty to weigh losses for scene geometry and semantics\.InProceedings of CVPR 2018,pp\. 7482–7491\.Cited by:[§3\.2](https://arxiv.org/html/2610.38044#S3.SS2.SSSx4.p1.1)\.
- \[5\]Y\. Cui and et al\.\(2019\)Class\-balanced loss based on effective number of samples\.InProceedings of CVPR 2019,pp\. 9260–9269\.Cited by:[§3\.2](https://arxiv.org/html/2610.38044#S3.SS2.SSSx2.p1.1)\.
- \[6\]DeepSeek\-AI and Unsloth\(2024\)DeepSeek\-R1\-Distill\-Llama\-8B\.Note:\[Model\] Hugging FaceAccessed: 21 Mar 2025Cited by:[§3\.3](https://arxiv.org/html/2610.38044#S3.SS3.p1.1)\.
- \[7\]D\. Demszky and et al\.\(2020\)GoEmotions: a dataset of fine\-grained emotions\.InProceedings of ACL 2020,pp\. 4040–4054\.Cited by:[§3\.2](https://arxiv.org/html/2610.38044#S3.SS2.SSSx2.p1.1),[§3\.5](https://arxiv.org/html/2610.38044#S3.SS5.p1.1),[Table 2](https://arxiv.org/html/2610.38044#S3.T2.1.3.1)\.
- \[8\]M\. A\. Gumilanget al\.\(2024\)Development of a mental health chatbot using large language models for indonesian undergraduates\.InProceedings of the 2024 IC3INA,Vol\.10732708,pp\. 337–342\.Cited by:[Table 1](https://arxiv.org/html/2610.38044#S2.T1.1.4.1)\.
- \[9\]R\. Islam and S\. W\. Bae\(2024\)Revolutionizing mental health support: An innovative affective mobile framework for dynamic, proactive, and context\-adaptive conversational agents\.CoRRabs/2406\.15942\.Cited by:[§1](https://arxiv.org/html/2610.38044#S1.p2.1)\.
- \[10\]S\. Jiet al\.\(2021\)MentalBERT: publicly available pretrained language models for mental healthcare\.CoRRabs/2110\.15621\.Cited by:[Table 1](https://arxiv.org/html/2610.38044#S2.T1.1.5.1),[§3\.2](https://arxiv.org/html/2610.38044#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.38044#S4.SS1.p1.1)\.
- \[11\]S\. Jiet al\.\(2022\)Suicidal ideation and mental disorder detection with attentive relation networks\.Neural Computing & Applications34\(13\),pp\. 10309–10319\.Cited by:[§3\.2](https://arxiv.org/html/2610.38044#S3.SS2.SSSx1.p1.1),[§3\.5](https://arxiv.org/html/2610.38044#S3.SS5.p1.1),[Table 2](https://arxiv.org/html/2610.38044#S3.T2.1.2.1)\.
- \[12\]Mistral AI\(2024\)Mistral\-7B\-Instruct\-v0\.3\.Note:Hugging FaceAccessed: Apr 2025Cited by:[§3\.3](https://arxiv.org/html/2610.38044#S3.SS3.p1.1),[Table 5](https://arxiv.org/html/2610.38044#S4.T5.1.1.3.1)\.
- \[13\]Mistral AI\(2024\)Mixtral\-8x7B\-Instruct\-v0\.1\.Note:Hugging FaceAccessed: May 2025Cited by:[Table 5](https://arxiv.org/html/2610.38044#S4.T5.1.1.2.1)\.
- \[14\]S\. M\. Mohammad\(2025\)NRC VAD Lexicon v2: norms for valence, arousal, and dominance for over 55k english terms\.CoRRabs/2503\.23547\.Cited by:[§3\.5](https://arxiv.org/html/2610.38044#S3.SS5.p2.1),[Table 2](https://arxiv.org/html/2610.38044#S3.T2.1.4.1)\.
- \[15\]H\. Rashkin and et al\.\(2019\)Towards empathetic open\-domain conversation models: A new benchmark and dataset\.InProceedings of ACL2019,pp\. 5370–5381\.Cited by:[§3\.5](https://arxiv.org/html/2610.38044#S3.SS5.p2.1),[Table 2](https://arxiv.org/html/2610.38044#S3.T2.1.6.1)\.
- \[16\]K\. Yanget al\.\(2024\)MentalLLaMA: interpretable mental health analysis on social media with large language models\.InProc\. of WWW 2024,pp\. 4489–4500\.Cited by:[Table 1](https://arxiv.org/html/2610.38044#S2.T1.1.2.1),[§3\.6](https://arxiv.org/html/2610.38044#S3.SS6.SSSx2.p1.1),[Table 5](https://arxiv.org/html/2610.38044#S4.T5.1.1.4.1)\.
- \[17\]A\. Yuanet al\.\(2025\)Improving workplace well\-being in modern organizations: A review of large language model\-based mental health chatbots\.ACM Trans\. Manag\. Inf\. Syst\.16\(1\)\.Cited by:[§1](https://arxiv.org/html/2610.38044#S1.p2.1),[Table 1](https://arxiv.org/html/2610.38044#S2.T1.1.3.1),[§2](https://arxiv.org/html/2610.38044#S2.p2.1)\.