Interpretable Depression Detection from Social Media Text Using LLM-Derived Embeddings

# Interpretable Depression Detection from Social Media Text Using LLM-Derived Embeddings
Source: [https://arxiv.org/html/2506.06616](https://arxiv.org/html/2506.06616)
11institutetext:Department of Computer Science, Earlham College, Richmond, IN, USA
11email:skim24@earlham\.edu; ooimieye23@earlham\.edu22institutetext:Department of Computer Science, Eastern Michigan University, Ypsilanti, MI, USA
22email:yyin@emich\.edu
\*Corresponding author###### Abstract

Accurate and interpretable detection of depressive language in social media can support early identification of mental health conditions and inform timely interventions\. In this paper, we investigate the use of large language models \(LLMs\) and traditional machine learning classifiers for three social media\-based mental health prediction tasks: binary depression classification, depression severity classification, and differential diagnosis among depression, PTSD, and anxiety\. We compare zero\-shot LLMs with supervised classifiers trained on conventional text embeddings, psycholinguistic features, and embeddings derived from LLM\-generated mental health summaries\. Across multiple publicly available social media text datasets and five\-fold cross\-validation experiments, we find that zero\-shot LLMs exhibit strong performance and generalization in binary depression classification, but struggle with fine\-grained severity prediction\. In contrast, supervised models trained on LLM summary embeddings often achieve more accurate and consistent performance, particularly for multi\-class and ordinal classification tasks\. These findings highlight both the strengths and limitations of current LLMs for mental health prediction and suggest that using LLMs as semantic interpreters, rather than solely as end\-to\-end classifiers, may provide a promising direction for building more effective and interpretable mental health assessment systems\.

## 1Introduction

Mental health disorders such as depression affect hundreds of millions of individuals worldwide, with many cases remaining undiagnosed or untreated due to social stigma, cost, or lack of access to care\. As individuals increasingly express their thoughts and emotions on social media, these platforms have become a valuable source of real\-time data for assessing psychological well\-being\. Automatic detection of depressive language from social media posts can be a promising tool for large\-scale, low\-cost mental health screening and intervention\. Prior approaches to depression classification have typically relied on two types of features: psycholinguistic markers and text embeddings derived from pretrained language models\. While psycholinguistic features such as those derived from the Linguistic Inquiry and Word Count \(LIWC\) lexicon offer interpretability, they are limited in expressiveness\. On the other hand, traditional sentence embeddings capture rich semantic information but may lack the specific affective cues that are critical for mental health prediction tasks\.

In this work, we propose a novel prompt\-based embedding approach that makes use of the reasoning capabilities of LLMs to produce embeddings with more interpretability and semantic richness\. Instead of directly embedding the raw input text, we prompt an LLM with a mental health\-oriented question and feed it with users’ social media post\. We then extract embedding from this LLM summary using a sentence encoder and use it as input to classifiers\. This method induces reasoning by forcing the LLM to generate a semantically enriched interpretation beyond surface\-level syntax\. It also reduces noise by allowing the LLM to filter irrelevant information\. Moreover, it enhances interpretability by producing intermediate summaries that can be shown to clinicians as part of an intervention or diagnosis\. We evaluate our approach on five social media\-based depression datasets and find that LLM\-derived summary embeddings, while not consistently superior, often enhance predictive performance over models using raw text embeddings in supervised tasks, and outperform a pre\-trained large language model in ordinal multi\-class classification\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2506.06616#S2)presents a review of related work, including traditional text\-based approaches for mental health prediction from social media and recent applications of large language models in this domain\. Section[3](https://arxiv.org/html/2506.06616#S3)introduces our methodology, including data preprocessing, feature extraction using text embeddings and psycholinguistic features, and the generation of LLM\-based summary embeddings\. Section[4](https://arxiv.org/html/2506.06616#S4)reports experimental results across three classification tasks: binary depression classification, depression severity classification, and differential diagnosis among depression, anxiety, and PTSD\. Finally, Section[5](https://arxiv.org/html/2506.06616#S5)concludes the paper with a discussion of key findings and future directions\.

## 2Related Works

### 2\.1Language\-Based Mental Health Assessment

Textual data, whether derived from written language, transcribed speech, or online interactions, offers a powerful lens into mental health, with numerous studies demonstrating that linguistic patterns can reflect emotional states and clinical symptoms associated with mental disorders\. Stress detection has been explored using a range of textual sources, including online blog and forum posts\[[1](https://arxiv.org/html/2506.06616#bib.bib1),[2](https://arxiv.org/html/2506.06616#bib.bib2),[3](https://arxiv.org/html/2506.06616#bib.bib3)\]and social media interactions\[[4](https://arxiv.org/html/2506.06616#bib.bib4),[5](https://arxiv.org/html/2506.06616#bib.bib5),[6](https://arxiv.org/html/2506.06616#bib.bib6),[7](https://arxiv.org/html/2506.06616#bib.bib7)\]\. PTSD diagnosis has also been performed using clinical patient narratives\[[8](https://arxiv.org/html/2506.06616#bib.bib8),[9](https://arxiv.org/html/2506.06616#bib.bib9)\], online surveys\[[10](https://arxiv.org/html/2506.06616#bib.bib10)\], and transcribed voicemails\[[11](https://arxiv.org/html/2506.06616#bib.bib11)\]\. Sawalha et al\.\[[12](https://arxiv.org/html/2506.06616#bib.bib12)\]argue that sentiment analysis of transcribed text from semi\-structured virtual interviews can effectively identify individuals with PTSD using a Random Forest model with VADER sentiment scores\. Mansoor et al\.\[[13](https://arxiv.org/html/2506.06616#bib.bib13)\]introduce a multimodal AI model that analyzes multilingual social media data to detect early signs of mental health crises, and emphasize the need for ethical safeguards and culturally sensitive applications in real\-world mental health systems\. Althoff et al\.\[[14](https://arxiv.org/html/2506.06616#bib.bib14)\]present a large\-scale quantitative analysis of text\-message\-based counseling conversations using computational discourse methods\. Ewbank et al\.\[[15](https://arxiv.org/html/2506.06616#bib.bib15)\]develop a deep learning model to automatically categorize patient utterances during internet\-enabled cognitive behavioral therapy\. Bantilan et al\.\[[16](https://arxiv.org/html/2506.06616#bib.bib16)\]propose an NLP model to detect suicide risk in patient messages during teletherapy, using therapist intervention patterns and expert annotations to label risk levels\. These studies highlight the growing potential of natural language processing \(NLP\) for mental health assessment and intervention, and show the importance of contextual and linguistic features in the real\-world applicability of such models\.

### 2\.2Large Language Models for Mental Health Prediction

Recent advancements in large language models have enabled their application across a wide range of domains, including the analysis and prediction of mental health conditions\. Xu et al\.\[[17](https://arxiv.org/html/2506.06616#bib.bib17)\]evaluates the performance of several large language models on mental health prediction tasks using online text data\. Their findings suggest that while zero\- and few\-shot prompting yield limited results, instruction fine\-tuning significantly boosts accuracy\. Boggavarapu et al\.\[[18](https://arxiv.org/html/2506.06616#bib.bib18)\]explore the use of LLMs enhanced with Retrieval\-Augmented Generation to predict mental health\-related ICD\-10\-CM codes from clinical notes, and find that current LLMs still struggle with accurately interpreting these complex codes\. Their findings suggest the need for better integration of structured medical knowledge into these models\. Malgaroli et al\.\[[19](https://arxiv.org/html/2506.06616#bib.bib19)\]discuss the potential of LLMs to advance mental health care through improved diagnostics, monitoring, and treatment\. They also identify challenges such as bias, accessibility, and data representation\. Qian el al\.\[[20](https://arxiv.org/html/2506.06616#bib.bib20)\]explore how foundation models, such as LLMs, are transforming digital mental health through personalized diagnostics, real\-time monitoring, emotion recognition, and adaptive interventions using multimodal data\. They propose a sociotechnical framework that integrates brain\-inspired AI, and clinical oversight with ethical considerations\. Hua et al\.\[[21](https://arxiv.org/html/2506.06616#bib.bib21)\]review the current landscape of LLM applications in mental health care, and conclude that there is promising use cases in counseling and clinical support, but most studies lack standardized evaluation methods\. Together, these studies show the growing potential of LLMs in mental health diagnosis and care\. Addressing the challenges in model reliability, interpretability, and evaluation rigor will be crucial for the integration of LLMs into real\-world clinical settings\.

### 2\.3LLMs as Reasoning and Summarization Tools

Beyond direct classification, large language models can reason over context and summarize complex narratives to generate structured representations that support downstream machine learning tasks\. SemCSE\[[22](https://arxiv.org/html/2506.06616#bib.bib22)\]was proposed as an unsupervised embedding method that uses LLM\-generated summaries of scientific abstracts as semantically enriched training signals, showing that summary\-based representations can improve semantic embedding quality for downstream tasks\. Liu et al\.\[[23](https://arxiv.org/html/2506.06616#bib.bib23)\]find that embeddings derived from LLM\-generated descriptions of genes and cell metadata can serve as effective semantic representations to improving performance on downstream biological analysis tasks\. Moreover, LLMs can be used as intermediate representation generators\. Li and Zhou propose a training\-free embedding approach\[[24](https://arxiv.org/html/2506.06616#bib.bib24)\]that combines mixture\-of\-experts routing weights with hidden\-state representations, showing that internal LLM activations can also provide useful semantic representations for downstream embedding tasks\. These works motivate our exploration of summary\-based embeddings as a mechanism for combining the semantic reasoning capabilities of LLMs with the transparency and efficiency of traditional classifiers\.

## 3Methods

Our methodology is organized into five main stages: data preprocessing, text embedding and psycholinguistic feature extraction, zero\-shot LLM prompting, LLM summary embedding generation, and model training and evaluation\.

### 3\.1Data Preprocessing

We preprocess five publicly available social media\-based mental health datasets for our experiments: MHB\[[25](https://arxiv.org/html/2506.06616#bib.bib25)\], CAMS\[[26](https://arxiv.org/html/2506.06616#bib.bib26)\], HelaDepDet\[[27](https://arxiv.org/html/2506.06616#bib.bib27)\], RMHD\[[28](https://arxiv.org/html/2506.06616#bib.bib28)\], and DepressionEmo\[[29](https://arxiv.org/html/2506.06616#bib.bib29)\]\. Each dataset consists of short, user\-generated text entries annotated with mental health labels, primarily related to depression\. To enable evaluation of the model’s ability to distinguish between depressive and non\-depressive language, we include two general\-domain Reddit control datasets, RedditAITA\[[30](https://arxiv.org/html/2506.06616#bib.bib30)\]and RedditTIFU\[[32](https://arxiv.org/html/2506.06616#bib.bib32)\], as non\-depression examples in the combined dataset\. These two datasets were selected because they are publicly available corpora containing informal first\-person narratives that are stylistically similar to the social media posts found in the mental health datasets\. To maintain a balanced representation between the two control sources, RedditTIFU was sampled to contain a comparable number of instances as RedditAITA\.

Table 1:Statistics of Datasets Used Across Classification TasksDatasetTaskSize\(posts\)Avg\.wordsLabels\(%\)MHB\[[25](https://arxiv.org/html/2506.06616#bib.bib25)\]BinaryDifferential7,452253Depression: 43Anxiety: 45PTSD: 12CAMS\[[26](https://arxiv.org/html/2506.06616#bib.bib26)\]Binary4,042179All depressionHelaDepDet\[[27](https://arxiv.org/html/2506.06616#bib.bib27)\]BinarySeverity33,498120Minimum: 25Mild: 25Moderate: 23Severe: 27RMHD\[[28](https://arxiv.org/html/2506.06616#bib.bib28)\]BinaryDifferential658236Depression: 34Anxiety: 25Other: 41DepressionEmo\[[29](https://arxiv.org/html/2506.06616#bib.bib29)\]Binary4,83095All depressionRedditAITA\[[30](https://arxiv.org/html/2506.06616#bib.bib30)\]Binary24,795386All non\-depression controlRedditTIFU\[[32](https://arxiv.org/html/2506.06616#bib.bib32)\]Binary25,685341All non\-depression controlTotal \(unique\)—100,960——We begin by cleaning each dataset through a series of preprocessing steps\. Duplicate entries are removed, and we retain only posts with text lengths falling between the 10th and 90th percentiles to exclude outliers\. For each dataset, only the columns relevant to the downstream tasks are preserved, while metadata fields such as text length, number of posts by the author, and number of likes are removed\. Our experimental setup supports three classification tasks:

1. 1\.Binary Depression Classification:We combine all five depression\-related datasets with RedditAITA and RedditTIFU non\-depression controls to train models that distinguish between depressive and non\-depressive content\.
2. 2\.Depression Severity Classification:Using the HelaDepDet\[[27](https://arxiv.org/html/2506.06616#bib.bib27)\]dataset, which provides graded depression severity labels, we train models to predict levels including minimum, mild, moderate, and severe\.
3. 3\.Differential Diagnosis Classification:We use the MHB\[[25](https://arxiv.org/html/2506.06616#bib.bib25)\]and RMHD\[[28](https://arxiv.org/html/2506.06616#bib.bib28)\]datasets, which contain multi\-class annotations for depression, anxiety, and PTSD, to assess the model’s ability to differentiate between related mental health conditions\.

Descriptive statistics for the preprocessed datasets, including the number of posts, label categories, and average text length measured in words, are summarized in Table[1](https://arxiv.org/html/2506.06616#S3.T1)\. For supervised experiments using text embeddings and machine learning classifiers, we applied deterministic five\-fold stratified cross\-validation\. Each task dataset was partitioned into five folds while preserving the class distribution within each fold\. During each iteration, models were trained on four folds and evaluated on the remaining held\-out fold\. Notably, two zero\-shot LLM\-based classification approaches do not require model training\. These methods were evaluated directly on the full task datasets using prompting strategies without parameter updates\.

### 3\.2Text Embedding and Psycholinguistic Feature Extraction

To establish baseline performance, we evaluate classifiers trained on the combination of two types of traditional feature representations:

#### 3\.2\.1Text Embeddings

We extract contextualized sentence\-level embeddings using theall\-mpnet\-base\-v2\[[31](https://arxiv.org/html/2506.06616#bib.bib31)\]model from the SentenceTransformers library\. Each social media post is passed through the pretrained model to obtain a 768\-dimensional fixed\-size embedding vector\.

#### 3\.2\.2Psycholinguistic Features

We compute psycholinguistic features using the Linguistic Inquiry and Word Count \(LIWC\) lexicon\. Each post is analyzed to yield normalized frequencies of relevant categories, including affective processes, cognitive processes, and pronoun usage\. The resulting feature vectors are standardized using z\-score normalization\.

#### 3\.2\.3Classification Models

We evaluate three traditional machine learning classifiers using both text embeddings alone and the concatenation of text embeddings with psycholinguistic features\. Logistic Regression was configured with L2 regularization \(C=1\.0C=1\.0\) and a maximum of 1,000 iterations\. The Support Vector Machine \(SVM\) employed a linear kernel withC=1\.0C=1\.0, while the Random Forest classifier consisted of 100 decision trees\. All supervised experiments were conducted using deterministic five\-fold stratified cross\-validation\. Performance was evaluated using accuracy, precision, recall, macro\-F1, and class\-wise F1 where applicable, and we report the mean and standard deviation across the five folds\.

### 3\.3Zero\-Shot LLM\-Based Classification

To evaluate large language models as direct zero\-shot classifiers, we use the OpenAI GPT\-4o and GPT\-5\.5 API\. For each social media post, we send a prompt requesting a binary or multi\-class classification label, depending on the task setting\. The prompt format for the binary classification task is as follows:

> You are a mental health expert\. Read the following social media post and determine the user’s mental health condition\. Choose from the following labels: Depression, Non\-depression\.

For severity detection and differential diagnosis, the label options are modified accordingly\. The model’s textual response is parsed as the predicted label\. No training or fine\-tuning is applied\.

### 3\.4Prompted LLM Summary Embedding

In this approach, we prompt GPT\-5\.5, the best\-performing LLM in preliminary experiments, to interpret and summarize the user’s mental state based on the content of a social media post\. The goal is to produce a concise, clinically oriented description that captures signals relevant to mental health assessment\. The prompt format for the LLM summary generation is as follows:

> You are a mental health expert\. Read the following social media post and describe the user’s mental state in one or two sentences\. Focus on emotional tone, cognitive state, and any signs of mental health conditions\. Avoid quoting the post verbatim\.

We embed the LLM\-generated response using theall\-mpnet\-base\-v2\[[31](https://arxiv.org/html/2506.06616#bib.bib31)\]sentence embedding model, resulting in a 768\-dimensional feature vector\. This vector captures task\-relevant affective semantics abstracted from the original text\. We then train the same set of classifiers described in Section[3\.2\.3](https://arxiv.org/html/2506.06616#S3.SS2.SSS3)on these embeddings to assess whether LLM\-generated paraphrased representations improve predictive performance\.

An additional advantage of this approach is interpretability\. Traditional sentence embeddings encode semantic information in a high\-dimensional latent space that is difficult for humans to inspect directly\. With an intermediate summary, the generated prediction by downstream classifiers has a transparent representation that can be reviewed by clinicians or other stakeholders to better understand the information being used\.

### 3\.5Evaluation Metrics

Results are reported as the mean and standard deviation across five\-fold cross\-validation\. For the binary depression classification task, we report accuracy, precision, recall, and F1\-score\. For the depression severity detection and differential diagnosis tasks, we report accuracy, macro F1, and class\-wise F1\-scores to provide a more comprehensive assessment of performance across categories\. All metrics are computed using thescikit\-learn\[[33](https://arxiv.org/html/2506.06616#bib.bib33)\]library\.

![Refer to caption](https://arxiv.org/html/2506.06616v2/figures/task1_metrics_updated.png)Figure 1:Binary classification accuracy and F1\-score across feature representations and classifiers\. Error bars show cross\-validation standard deviation\.

## 4Experiment Results

In this section, we present the evaluation results across three mental health classification tasks\. Each task is designed to assess the model’s ability to handle different types of diagnoses\. We begin with binary depression classification, where the goal is to distinguish between depressive and non\-depressive content using a combination of datasets\. Next, we examine depression severity classification, in which the models predict fine\-grained severity levels using the HelaDepDet\[[27](https://arxiv.org/html/2506.06616#bib.bib27)\]dataset\. Finally, we evaluate the performance of the models and embeddings on the task of differential diagnosis classification among depression, anxiety, and PTSD, using the MHB\[[25](https://arxiv.org/html/2506.06616#bib.bib25)\]and RMHD\[[28](https://arxiv.org/html/2506.06616#bib.bib28)\]datasets\.

Table 2:Binary classification metrics by model and feature type\. Cross\-validation rows report five\-fold mean±\\pmstandard deviation\.ModelAccuracyPrecisionRecallF1\-scoreLogistic Regression\(MPNet\)0\.838±\\pm0\.0010\.840±\\pm0\.0010\.835±\\pm0\.0010\.837±\\pm0\.002SVM\(MPNet\)0\.862±\\pm0\.0030\.866±\\pm0\.0020\.858±\\pm0\.0030\.862±\\pm0\.003Random Forest\(MPNet\)0\.854±\\pm0\.0020\.863±\\pm0\.0020\.846±\\pm0\.0030\.854±\\pm0\.002Logistic Regression\(MPNet \+ LIWC\)0\.894±\\pm0\.0020\.938±\\pm0\.0010\.890±\\pm0\.0020\.904±\\pm0\.002SVM\(MPNet \+ LIWC\)0\.879±\\pm0\.0010\.901±\\pm0\.0010\.876±\\pm0\.0010\.898±\\pm0\.001Random Forest\(MPNet \+ LIWC\)0\.907±\\pm0\.0020\.955±\\pm0\.0010\.899±\\pm0\.0020\.937±\\pm0\.002GPT\-4o Zero\-Shot0\.9620\.9700\.9680\.965GPT\-5\.5 Zero\-Shot0\.9800\.9750\.9710\.969Logistic Regression\(LLM Summary\)0\.886±\\pm0\.0010\.890±\\pm0\.0000\.884±\\pm0\.0020\.887±\\pm0\.001SVM\(LLM Summary\)0\.861±\\pm0\.0000\.862±\\pm0\.0010\.859±\\pm0\.0010\.861±\\pm0\.000Random Forest\(LLM Summary\)0\.878±\\pm0\.0010\.887±\\pm0\.0010\.869±\\pm0\.0010\.878±\\pm0\.001### 4\.1Binary Depression Classification

For the binary classification task of determining whether a social media post is depressive or non\-depressive, performance metrics are summarized in Table[2](https://arxiv.org/html/2506.06616#S4.T2)and shown in Figure[1](https://arxiv.org/html/2506.06616#S3.F1)\. Among all models evaluated, the zero\-shot GPT\-5\.5 classifier achieved the highest overall accuracy, outperforming both traditional machine learning models using psycholinguistic and text\-based embeddings, as well as models predicting based on LLM\-generated summary embeddings\.

![Refer to caption](https://arxiv.org/html/2506.06616v2/figures/figure3_cubehelix_palette.png)Figure 2:Class\-wise F1\-scores for depression severity classification\. Error bars indicate standard deviations across five\-fold cross\-validation for supervised models\.Notably, the zero\-shot LLM was not specifically fine\-tuned for depression detection, yet demonstrated strong performance, likely due to its extensive pretraining on large\-scale, diverse datasets\. This proves the model’s impressive generalization capabilities and supports recent findings on the excellent performance of LLMs in zero\-shot settings\.

Machine learning models using LLM\-generated summary embeddings performed better than those using features extracted directly from the raw social media text\. This outcome is expected as the summaries provide a condensed, higher\-level interpretation of each post, which likely makes implicit depressive cues more accessible\.

### 4\.2Depression Severity Classification

![Refer to caption](https://arxiv.org/html/2506.06616v2/figures/task2_group2.png)
![Refer to caption](https://arxiv.org/html/2506.06616v2/figures/task2_group3.png)

Figure 3:Violin plots of predicted depression severity grouped by the ground\-truth, illustrating the separation achieved by a zero\-shot LLM \(left\) and a supervised classifier using LLM\-summary embeddings \(right\)\.To evaluate how well different models and embeddings can assess the severity of depression from users’ social media posts, we conducted a multi\-class classification task using HelaDepDet\[[27](https://arxiv.org/html/2506.06616#bib.bib27)\]dataset\. This dataset includes four ordinal labels representing increasing levels of depression severity, ranging from 0 \(minimum\) to 3 \(severe\)\.

We applied the same classification framework as in the previous task, comparing both zero\-shot LLM\-based classification and traditional machine learning models with various feature representations\. Among all methods, the Logistic Regression classifier using LLM\-generated summary embeddings achieved the highest accuracy of 58%, slightly outperforming models using alternative features, and significantly outperforming the zero\-shot LLM\-based approach\.

Figure[2](https://arxiv.org/html/2506.06616#S4.F2)shows the per\-class F1 scores for each classifier\. Figure[3](https://arxiv.org/html/2506.06616#S4.F3)present violin plots comparing predicted and true severity levels for the worst and best\-performing models\. We observe that the zero\-shot LLM classifier struggles to infer fine\-grained severity levels directly from raw text, often failing to reflect the ordinal structure of the labels\. In contrast, trained machine learning models benefit from supervised learning, capturing both the semantic and ordinal relationships in the data, and are able to make accurate and consistent depression severity predictions\.

Table 3:Differential diagnosis classification across feature representations and modelsModelAcc\.MacroF1AnxietyF1Depress\.F1PTSDF1Logistic Regression\(MPNet\)0\.769±\\pm0\.0040\.765±\\pm0\.0070\.753±\\pm0\.0130\.751±\\pm0\.0060\.791±\\pm0\.005SVM\(MPNet\)0\.781±\\pm0\.0100\.770±\\pm0\.0020\.749±\\pm0\.0080\.775±\\pm0\.0090\.786±\\pm0\.003Random Forest\(MPNet\)0\.754±\\pm0\.0090\.730±\\pm0\.0120\.674±\\pm0\.0320\.754±\\pm0\.0060\.763±\\pm0\.010Logistic Regression\(MPNet \+ LIWC\)0\.776±\\pm0\.0050\.773±\\pm0\.0060\.757±\\pm0\.0090\.769±\\pm0\.0030\.793±\\pm0\.007SVM\(MPNet \+ LIWC\)0\.802±\\pm0\.0040\.804±\\pm0\.0050\.800±\\pm0\.0030\.799±\\pm0\.0050\.814±\\pm0\.008Random Forest\(MPNet \+ LIWC\)0\.769±\\pm0\.0030\.731±\\pm0\.0080\.691±\\pm0\.0130\.732±\\pm0\.0110\.769±\\pm0\.002GPT\-4o Zero\-Shot0\.8750\.8550\.8250\.8680\.873GPT–5\.5Zero\-Shot0\.8910\.8760\.8310\.9020\.895Logistic Regression\(LLM Summary\)0\.785±\\pm0\.0080\.776±\\pm0\.0060\.744±\\pm0\.0100\.781±\\pm0\.0050\.803±\\pm0\.006SVM\(LLM Summary\)0\.826±\\pm0\.0050\.819±\\pm0\.0050\.823±\\pm0\.0110\.794±\\pm0\.0020\.840±\\pm0\.003Random Forest\(LLM Summary\)0\.772±\\pm0\.0040\.744±\\pm0\.0070\.720±\\pm0\.0080\.734±\\pm0\.0090\.777±\\pm0\.003
### 4\.3Differential Diagnosis Classification

We evaluated model performance on the task of differential diagnosis classification using the MHB\[[25](https://arxiv.org/html/2506.06616#bib.bib25)\]and RMHD\[[28](https://arxiv.org/html/2506.06616#bib.bib28)\]datasets, which include multi\-class annotations for depression, PTSD, and anxiety\. This task assesses the models’ ability to distinguish between related but clinically distinct mental health conditions based on social media text\.

Among all evaluated methods, the zero\-shot GPT\-5\.5 and GPT\-4o classifiers achieved the highest overall accuracies, slightly outperforming the summary\-based embedding approaches, while traditional machine learning models trained on text embeddings combined with LIWC features generally produced the lowest overall accuracies among the evaluated methods\. Table[3](https://arxiv.org/html/2506.06616#S4.T3)shows the F1 score distribution across all classifiers and diagnostic classes\. We observe that that most classifiers tend to confuse depression with anxiety, which is expected as the two mental health conditions have overlapping linguistic and emotional patterns, and social media expressions can reflect that\. Moreover, PTSD is more consistently distinguished, possibly due to more specific symptom language, such as references to trauma, that sets it apart from the other two\.

## 5Conclusion and Future Work

This study presents a comparative evaluation of zero\-shot LLMs and traditional machine learning models in depression classification tasks based on social media data\. We find that zero\-shot LLMs have strong performance in binary depression classification, proving their ability to generalize from pretrained knowledge\. However, their performance declines in tasks such as severity prediction, where supervised models with LLM\-generated summary embeddings show more accurante and consistent performance\.

Our evaluation results suggest that LLMs are powerful for mental health prediction tasks, and their contextual summaries are helpful to derive better features\. Summary embeddings derived from LLMs capture important semantic cues and can improve traditional models to make more accurate and consistent predictions\. Summaries can also serve as human\-readable explanations of the signals contributing to a prediction and may help stakeholders assess whether a model’s conclusions are supported by meaningful linguistic evidence\. These findings prove the potential of hybrid approaches that combine the generalization capabilities of LLMs with lightweight, interpretable classifiers trained on curated features\. Further performance gains may be achieved by exploring advanced prompting strategies, applying few\-shot learning, and fine\-tuning LLMs on domain\-specific mental health data\.

\{credits\}

#### 5\.0\.1Acknowledgements

This research was supported by the Lemann Student/Faculty Collaborative Research Fund at Earlham College\. We gratefully acknowledge their funding and support\.

#### 5\.0\.2\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]Inamdar, S\., Chapekar, R\., Gite, S\., Pradhan, B\.: Machine learning driven mental stress detection on reddit posts using natural language processing\. Human\-Centric Intelligent Systems 3\(2\), 80–91 \(Jun 2023\)\.[https://doi\.org/10\.1007/s44230\-023\-00020\-8](https://doi.org/10.1007/s44230-023-00020-8)
- \[2\]Zhao, L\., Jia, J\., Feng, L\.: Teenagers’ stress detection based on time\-sensitive micro\-blog comment/response actions\. In: Dillon, T\. \(ed\.\) Artificial Intelligence in Theory and Practice IV\. pp\. 26–36\. Springer International Publishing, Cham \(2015\)
- \[3\]Desmet, B\., Jacobs, G\., Hoste, V\.: Mental distress detection and triage in forum posts: The LT3 CLPsych 2016 shared task system\. In: Hollingshead, K\., Ungar, L\. \(eds\.\) Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology\. pp\. 148–152\. Association for Computational Linguistics, San Diego, CA, USA \(Jun 2016\)\.[https://aclanthology\.org/W16\-0317/](https://aclanthology.org/W16-0317/)
- \[4\]Nijhawan, T\., Attigeri, G\., Ananthakrishna, T\.: Stress detection using natural language processing and machine learning over social interactions\. Journal of Big Data 9\(1\), 33 \(Mar 2022\)\.[https://doi\.org/10\.1186/s40537\-022\-00575\-6](https://doi.org/10.1186/s40537-022-00575-6)
- \[5\]Li, N\., Zhang, H\., Feng, L\.: Incorporating Forthcoming Events and Personality Traits in Social Media Based Stress Prediction\. IEEE Transactions on Affective Computing 14\(01\), 603\-621 \(Jan 2023\)\.[https://doi\.ieeecomputersociety\.org/10\.1109/TAFFC\.2021\.3076294](https://doi.ieeecomputersociety.org/10.1109/TAFFC.2021.3076294)
- \[6\]Guntuku, S\.C\., Buffone, A\., Jaidka, K\., Eichstaedt, J\.C\., Ungar, L\.H\.: Understanding and measuring psychological stress using social media\. In: Proceedings of the International AAAI Conference on Web and Social Media\. vol\. 13, pp\. 214–225 \(2019\)\.[https://doi\.org/10\.1609/icwsm\.v13i01\.3223](https://doi.org/10.1609/icwsm.v13i01.3223)
- \[7\]Kabir, M\.H\., Samrat, N\., Mahmud, A\.A\., Akter, R\., Raihan, M\.: Mental stress pre\- diction from the text of social media using machine learning techniques\. 2023 14th International Conference on Computing Communication and Networking Technolo\- gies \(ICCCNT\) pp\. 1–7 \(2023\)\.[https://api\.semanticscholar\.org/CorpusID:265406402](https://api.semanticscholar.org/CorpusID:265406402)
- \[8\]He, Q\., Veldkamp, B\., Glas, C\.: Automated assessment of patients’ self\-narratives for posttraumatic stress disorder screening using natural language processing and text mining\. Assessment 24, 157–172 \(03 2017\)\.[https://doi\.org/10\.1177/1073191115602551](https://doi.org/10.1177/1073191115602551)
- \[9\]Wshah, S\., Skalka, C\., Price, M\.: Predicting posttraumatic stress disorder risk: A machine learning approach\. JMIR Ment Health 6\(7\), e13946 \(Jul 2019\)\.[http://mental\.jmir\.org/2019/7/e13946/](http://mental.jmir.org/2019/7/e13946/)
- \[10\]Marengo, D\., Hoeboer, C\., Veldkamp, B\., Olff, M\.: Text mining to improve screening for trauma\-related symptoms in a global sample\. Psychiatry Research 316, 114753 \(2022\)\.[https://www\.sciencedirect\.com/science/article/pii/S016517812200347X](https://www.sciencedirect.com/science/article/pii/S016517812200347X)
- \[11\]Oltmanns, J\.R\., Schwartz, H\.A\., Ruggero, C\., Son, Y\., Miao, J\., Waszczuk, M\., Clouston, S\.A\., Bromet, E\.J\., Luft, B\.J\., Kotov, R\.: Artificial intelligence language predictors of two\-year trauma\-related outcomes\. Journal of Psychiatric Research 143,239–245 \(2021\)\.[https://www\.sciencedirect\.com/science/article/pii/S0022395621005549](https://www.sciencedirect.com/science/article/pii/S0022395621005549)
- \[12\]Sawalha, J\., Yousefnezhad, M\., Shah, Z\., Brown, M\.R\.G\., Greenshaw, A\.J\., Greiner, R\.: Detecting presence of ptsd using sentiment analysis from text data\. Frontiers in Psychiatry Volume 12 \- 2021 \(2022\)\.[https://www\.frontiersin\.org/journals/psychiatry/articles/10\.3389/fpsyt\.2021\.811392](https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2021.811392)
- \[13\]Mansoor, M\.A\., Ansari, K\.H\.: Early detection of mental health crises through artificial\-intelligence\-powered social media analysis: A prospective observational study\. Journal of Personalized Medicine 14\(9\) \(2024\)\.[https://www\.mdpi\.com/2075\-4426/14/9/958](https://www.mdpi.com/2075-4426/14/9/958)
- \[14\]Althoff, T\., Clark, K\., Leskovec, J\.: Large\-scale analysis of counseling conversations: An application of natural language processing to mental health\. Transactions of the Association for Computational Linguistics 4, 463–476 \(2016\)\.[https://aclanthology\.org/Q16\-1033/](https://aclanthology.org/Q16-1033/)
- \[15\]Ewbank, M\.P\., Cummins, R\., Tablan, V\., Catarino, A\., Buchholz, S\., Blackwell, A\.D\.: Understanding the relationship between patient language and outcomes in internet\-enabled cognitive behavioural therapy: A deep learning approach to automatic coding of session transcripts\. Psychotherapy Research 31\(3\), 300\-312 \(2021\)\.[https://doi\.org/10\.1080/10503307\.2020\.1788740](https://doi.org/10.1080/10503307.2020.1788740)
- \[16\]Bantilan, N\., Malgaroli, M\., Ray, B\., Hull, T\.D\.: Just in time crisis response: suicide alert system for telemedicine psychotherapy settings\. Psychotherapy Research 31\(3\), 289–299 \(2021\)\.[https://doi\.org/10\.1080/10503307\.2020\.1781952](https://doi.org/10.1080/10503307.2020.1781952)
- \[17\]Xu, X\., Yao, B\., Dong, Y\., Gabriel, S\., Yu, H\., Hendler, J\., Ghassemi, M\., Dey, A\.K\., Wang, D\.: Mental\-llm: Leveraging large language models for mental health prediction via online text data\. Proc\. ACM Interact\. Mob\. Wearable Ubiquitous Technol\. 8\(1\) \(Mar 2024\)\.[https://doi\.org/10\.1145/3643540](https://doi.org/10.1145/3643540)
- \[18\]Boggavarapu, L\., Srivastava, V\., Varanasi, A\.M\., Lu, Y\., Bhaumik, R\.: Evaluating enhanced llms for precise mental health diagnosis from clinical notes\. medRxiv \(2025\)\.[https://doi\.org/10\.1101/2024\.12\.16\.24317648](https://doi.org/10.1101/2024.12.16.24317648)
- \[19\]Malgaroli, M\., Schultebraucks, K\., Myrick, K\.J\., Andrade Loch, A\., Ospina\-Pinillos, L\., Choudhury, T\., Kotov, R\., De Choudhury, M\., Torous, J\.: Large language models for the mental health community: framework for translating code to care\. The Lancet Digital Health 7\(4\), e282–e285 \(Apr 2025\)\.[https://doi\.org/10\.1016/S2589\-7500\(24\)00255\-3](https://doi.org/10.1016/S2589-7500(24)00255-3)
- \[20\]Qian, K\., Zhang, H\., Jing, X\., Hu, B\., Yamamoto, Y\., Schuller, B\.W\.: Foundation models for digital mental health: igniting the dawn\. Medicine Plus 2\(2\), 100085 \(2025\)\.[https://www\.sciencedirect\.com/science/article/pii/S2950347725000167](https://www.sciencedirect.com/science/article/pii/S2950347725000167)
- \[21\]Hua, Y\., Na, H\., Li, Z\., Liu, F\., Fang, X\., Clifton, D\., Torous, J\.: A scoping review of large language models for generative tasks in mental health care\. npj Digital Medicine 8\(1\), 230 \(Apr 2025\)\.[https://doi\.org/10\.1038/s41746\-025\-01611\-4](https://doi.org/10.1038/s41746-025-01611-4)
- \[22\]Brinner, M\.F\., ZarrieSS, S\.: SemCSE: Semantic contrastive sentence embeddings using LLM\-generated summaries for scientific abstracts\. In: Christodoulopoulos, C\., Chakraborty, T\., Rose, C\., Peng, V\. \(eds\.\) Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing\. pp\. 32704–32719\. Association for Computational Linguistics, Suzhou, China \(Nov 2025\)\.[https://aclanthology\.org/2025\.emnlp\-main\.1662/](https://aclanthology.org/2025.emnlp-main.1662/)
- \[23\]Liu, T\., Chen, T\., Zheng, W\., Luo, X\., Chen, Y\., Zhao, H\.: Embeddings from language models are good learners for single\-cell data analysis\. Patterns 7\(2\) \(Feb 2026\)\.[https://doi\.org/10\.1016/j\.patter\.2025\.101431](https://doi.org/10.1016/j.patter.2025.101431)
- \[24\]Li, Z\., Zhou, T\.: Your mixture\-of\-experts llm is secretly an embedding model for free\. In: Yue, Y\., Garg, A\., Peng, N\., Sha, F\., Yu, R\. \(eds\.\) International Conference on Learning Representations\. vol\. 2025, pp\. 70029–70046 \(2025\),[https://proceedings\.iclr\.cc/paper\_files/paper/2025/file/aed2049f68827943dda5a63b7c4ba0a2\-Paper\-Conference\.pdf](https://proceedings.iclr.cc/paper_files/paper/2025/file/aed2049f68827943dda5a63b7c4ba0a2-Paper-Conference.pdf)
- \[25\]Boinepelli, S\., Raha, T\., Abburi, H\., Parikh, P\., Chhaya, N\., Varma, V\.: Leveraging mental health forums for user\-level depression detection on social media\. In: Calzolari, N\., Béchet, F\., Blache, P\., Choukri, K\., Cieri, C\., Declerck, T\., Goggi, S\., Isahara, H\., Maegaard, B\., Mariani, J\., Mazo, H\., Odijk, J\., Piperidis, S\. \(eds\.\) Proceedings of the Thirteenth Language Resources and Evaluation Conference\. pp\. 5418–5427\. European Language Resources Association, Marseille, France \(Jun 2022\),[https://aclanthology\.org/2022\.lrec\-1\.580/](https://aclanthology.org/2022.lrec-1.580/)
- \[26\]Garg, M\., Saxena, C\., Saha, S\., Krishnan, V\., Joshi, R\., Mago, V\.: CAMS: An annotated corpus for causal analysis of mental health issues in social media posts\. In: Calzolari, N\., Béchet, F\., Blache, P\., Choukri, K\., Cieri, C\., Declerck, T\., Goggi, S\., Isahara, H\., Maegaard, B\., Mariani, J\., Mazo, H\., Odijk, J\., Piperidis, S\. \(eds\.\) Proceedings of the Thirteenth Language Resources and Evaluation Conference\. pp\. 6387–6396\. European Language Resources Association, Marseille, France \(Jun 2022\),[https://aclanthology\.org/2022\.lrec\-1\.686/](https://aclanthology.org/2022.lrec-1.686/)
- \[27\]Priyadarshana, Y\.H\.P\.P\., Liang, Z\., Piumarta, I\.: Heladepdet: A novel multi\-class classification model for detecting the severity of human depression\. In: Takada, H\., Marutschke, D\.M\., Alvarez, C\., Inoue, T\., Hayashi, Y\., Hernandez\-Leo, D\. \(eds\.\) Collaboration Technologies and Social Computing\. pp\. 3–18\. Springer Nature Switzerland, Cham \(2023\)
- \[28\]Rani, S\., Ahmed, K\., Subramani, S\.: From posts to knowledge: Annotating a pandemic\-era reddit dataset to navigate mental health narratives\. Applied Sciences 14\(4\) \(2024\)\.[https://www\.mdpi\.com/2076\-3417/14/4/1547](https://www.mdpi.com/2076-3417/14/4/1547)
- \[29\]Rahman, A\.B\.S\., Ta, H\.T\., Najjar, L\., Azadmanesh, A\., Gönul, A\.S\.: Depressionemo: A novel dataset for multilabel classification of depression emotions\. Journal of Affective Disorders 366, 445–458 \(2024\)\.[https://www\.sciencedirect\.com/science/article/pii/S0165032724012278](https://www.sciencedirect.com/science/article/pii/S0165032724012278)
- \[30\]Alhassan, A\., Zhang, J\., Schlegel, V\.: “am I the bad one” ? predicting the moral judgement of the crowd using pre–trained language models\. In: Calzolari, N\., Béchet, F\., Blache, P\., Choukri, K\., Cieri, C\., Declerck, T\., Goggi, S\., Isahara, H\., Maegaard, B\., Mariani, J\., Mazo, H\., Odijk, J\., Piperidis, S\. \(eds\.\) Proceedings of the Thirteenth Language Resources and Evaluation Conference\. pp\. 267\-276\. European Language Resources Association, Marseille, France \(Jun 2022\),[https://aclanthology\.org/2022\.lrec\-1\.28/](https://aclanthology.org/2022.lrec-1.28/)
- \[31\]Song, K\., Tan, X\., Qin, T\., Lu, J\., Liu, T\.Y\.: Mpnet: masked and permuted pre\-training for language understanding\. In: Proceedings of the 34th International Conference on Neural Information Processing Systems\. NIPS ’20, Curran Associates Inc\., Red Hook, NY, USA \(2020\)
- \[32\]Kim, B\., Kim, H\., Kim, G\.: Abstractive summarization of Reddit posts with multi\-level memory networks\. In: Burstein, J\., Doran, C\., Solorio, T\. \(eds\.\) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)\. pp\. 2519–2531\. Association for Computational Linguistics, Minneapolis, Minnesota \(Jun 2019\)\.[https://aclanthology\.org/N19\-1260/](https://aclanthology.org/N19-1260/)
- \[33\]Pedregosa, F\., Varoquaux, G\., Gramfort, A\., Michel, V\., Thirion, B\., Grisel, O\., Blondel, M\., Prettenhofer, P\., Weiss, R\., Dubourg, V\., Vanderplas, J\., Passos, A\., Cournapeau, D\., Brucher, M\., Perrot, M\., Duchesnay, E\.: Scikit\-learn: Machine learning in python\. J\. Mach\. Learn\. Res\. 12\(null\), 2825–2830 \(Nov 2011\)