Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media

# Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media
Source: [https://arxiv.org/html/2608.26138](https://arxiv.org/html/2608.26138)
Rajveer Singh Pall and Sameer YadavR\. S\. Pall and S\. Yadav are with the Department of Computer Science and Business Systems, Gyan Ganga Institute of Technology and Sciences, Jabalpur, 482003, India\. E\-mail: \{rajveer\.pall, sameer\.yadav\}@ggits\.org Manuscript received April 26, 2026\.

###### Abstract

The reliability of mental health natural language processing \(NLP\) models across social media platforms remains insufficiently understood despite their growing deployment in real\-world settings\. We propose the Cross\-Platform Fairness Evaluation \(CPFE\) framework, a comprehensive five\-axis audit protocol assessing discriminative performance, calibration, statistical significance, prediction equity, and attribution stability\. Four transformer models \(BERT, RoBERTa, Emotion\-DistilRoBERTa, and GoEmotions\-RoBERTa\) were trained on a Kaggle mental health corpus \(n=35,556n=35\{,\}556\) and evaluated on independent Reddit \(n=6,257n=6\{,\}257\) and Twitter \(n=2,883n=2\{,\}883\) datasets, with emotion labels mapped to clinically relevant mental health proxy classes\. Across three independently evaluated models, macro AUC declined by 30\.3–35\.4% on Reddit and 37\.9–39\.5% on Twitter relative to in\-domain performance \(AUC = 0\.983–0\.987\)\. Calibration also deteriorated substantially, with expected calibration error increasing from 0\.056–0\.060 in\-domain to 0\.196–0\.229 on Reddit and 0\.499–0\.542 on Twitter\. Platform\-specific temperature scaling reduced calibration error by 88\.0% while preserving discrimination \(\|Δ​AUC\|<0\.01\|\\Delta\\mathrm\{AUC\}\|<0\.01\), demonstrating that calibration and discrimination represent distinct failure modes\. Fairness analysis revealed substantial cross\-platform prediction disparities, with equalized odds differences ranging from 0\.753 to 0\.831, whereas attribution stability analysis showed near\-complete divergence in influential vocabularies across platforms \(JaccardJ=0J=0in 14 of 16 model–class pairs atK=10K=10\)\. Finally, limited target\-domain fine\-tuning improved mean AUC by 0\.216, suggesting that even modest target\-platform supervision provides greater benefit than calibration alone\. These findings establish comprehensive cross\-platform evaluation as a fundamental prerequisite for the reliable, fair, and trustworthy deployment of mental health NLP systems\.

###### Index Terms:

mental healthnlp; cross\-platform generalisation; transformer models; prediction equity; calibration; domain shift; social media; disparate impact; temperature scaling; proxy labels\.

## 1Introduction

The proliferation of transformer\-based natural language processing \(nlp\) in clinical and near\-clinical settings has raised urgent questions about the conditions under which models trained on one data distribution remain reliable when deployed on another\. Mental healthnlprepresents a particularly high\-stakes instantiation of this problem\. Models that classify social media posts for proxy indicators of depression, anxiety, or stress are increasingly used in research pipelines and are proposed for deployment in crisis monitoring and digital mental health contexts\.

The present study focuses on*proxy\-label classifiers*— models that predict emotion\-derived indicators of mental health conditions rather than clinically validated diagnoses — and evaluates their cross\-platform reliability\. In such deployments, prediction errors carry downstream consequences: false negatives may delay human review of distress\-indicating language, while false positives at scale may contribute to alert fatigue\. We do not evaluate models in operational clinical settings; all findings are based on proxy emotion labels mapped to clinical categories\[[1](https://arxiv.org/html/2608.26138#bib.bib1),[2](https://arxiv.org/html/2608.26138#bib.bib2)\]\.

The challenge is compounded by the fragmented nature of social media data\. Platforms differ systematically in user demographics, communication norms, vocabulary, text length, and the ways in which users express psychological distress\[[3](https://arxiv.org/html/2608.26138#bib.bib3),[4](https://arxiv.org/html/2608.26138#bib.bib4)\]\. A model trained on labelled posts from one platform may encode platform\-specific stylistic features rather than signals that generalise to deployment contexts\.

Despite this risk, evaluation of mental healthnlphas largely been conducted in single\-platform settings\[[5](https://arxiv.org/html/2608.26138#bib.bib5),[6](https://arxiv.org/html/2608.26138#bib.bib6)\]\. Models are fine\-tuned and evaluated on splits of the same dataset, producing optimistic performance estimates that do not generalise to real deployment scenarios\. Rigorous cross\-platform evaluation protocols — analogous to external validation requirements in clinical prediction modelling\[[7](https://arxiv.org/html/2608.26138#bib.bib7),[8](https://arxiv.org/html/2608.26138#bib.bib8)\]— are absent from current practice\.

We address this gap by introducing the Cross\-Platform Fairness Evaluation \(cpfe\) framework, comprising five evaluation axes:

1. \(1\)discriminative performance under distribution shift \(auc, F1\);
2. \(2\)probabilistic calibration \(Expected Calibration Error,ece\);
3. \(3\)statistical significance of performance differences \(bootstrap\-corrected pairwiseauctests\);
4. \(4\)platform\-stratified prediction equity \(Symmetric Disparate Impact, Equalized Odds Difference\); and
5. \(5\)attribution stability \(Jaccard similarity of top\-KKfeature vocabularies\)\.

We applycpfeto four transformer models spanning distinct pretraining regimes — general\-domain BERT, general\-domain RoBERTa, an emotion\-fine\-tuned DistilRoBERTa, and a GoEmotions\-pretrained RoBERTa — trained on a combined Kaggle mental health corpus and evaluated on Reddit \(GoEmotions\[[9](https://arxiv.org/html/2608.26138#bib.bib9)\]\) and Twitter \(dair\-ai/emotion\[[10](https://arxiv.org/html/2608.26138#bib.bib10)\]\) test sets mapped to four clinical categories\.

### Principal Contributions

Prior cross\-platform work in mental healthnlp\[[12](https://arxiv.org/html/2608.26138#bib.bib12)\]established that performance degradation occurs under platform shift; the present study extends that line of inquiry by introducing a five\-axis framework that jointly characterises discriminative, calibration, equity, and attribution stability failures\. Specifically, we contribute:

1. \(i\)a unified five\-axiscpfeprotocol that jointly operationalises discriminative performance, probabilistic calibration, Bonferroni\-corrected statistical significance, platform\-stratified prediction equity, and gradient attribution stability as a structured audit framework for mental healthnlp;
2. \(ii\)evidence that all four evaluated models exhibit substantial cross\-platformaucdegradation \(30\.330\.3–39\.5%39\.5\\%;28\.628\.6–39\.5%39\.5\\%including the non\-independent GoEmotions\-RoBERTa Reddit result\), with gradient attribution stability approaching zero \(J=0J\{=\}0in14/1614/16model\-class pairs atK=10K\{=\}10\);
3. \(iii\)empirical validation that platform\-specific temperature scaling substantially recovers calibration \(meanecereduction88%88\\%\) while leaving discriminative performance unchanged; and
4. \(iv\)empirical reference ranges for cross\-platformaucdegradation \(30\.330\.3–39\.5%39\.5\\%for independently evaluated models;28\.628\.6–39\.5%39\.5\\%across all four models\) as a descriptive baseline for future cross\-platform evaluation work\.

## 2Related Work

### 2\.1Domain Shift in NLP

Distribution shift innlphas been studied in sentiment analysis\[[11](https://arxiv.org/html/2608.26138#bib.bib11)\], cross\-platform mental health classification\[[12](https://arxiv.org/html/2608.26138#bib.bib12)\], and clinical text mining\[[13](https://arxiv.org/html/2608.26138#bib.bib13)\]\. Foundational formalisations of dataset shift are provided by Quiñonero\-Candelaet al\.\[[26](https://arxiv.org/html/2608.26138#bib.bib26)\]\. Domain\-adaptive pretraining \(DAPT\)\[[14](https://arxiv.org/html/2608.26138#bib.bib14)\], adversarial domain adaptation, and self\-training on unlabelled target\-domain text have been proposed to mitigate shift, but are rarely evaluated in mental health settings\.

### 2\.2Mental Health NLP on Social Media

Transformer\-based classifiers for social\-media\-based depression, anxiety, and stress proxy indicators have demonstrated strong within\-platform discriminative performance\[[5](https://arxiv.org/html/2608.26138#bib.bib5),[6](https://arxiv.org/html/2608.26138#bib.bib6)\], though the clinical validity of such proxy labels remains contested\[[1](https://arxiv.org/html/2608.26138#bib.bib1)\]\. Chancellor and De Choudhury\[[1](https://arxiv.org/html/2608.26138#bib.bib1)\]provide a critical review noting widespread methodological limitations including lack of external validation, clinical label ambiguity, and selection bias\.

The most directly relevant prior work is Kulaet al\.\[[12](https://arxiv.org/html/2608.26138#bib.bib12)\], who studied cross\-platform generalisation of mental health content classifiers across Reddit and Twitter\. Our work extends Kulaet al\.\[[12](https://arxiv.org/html/2608.26138#bib.bib12)\]by adding probabilistic calibration, Bonferroni\-corrected significance testing, platform\-stratified prediction equity, and gradient attribution stability as evaluation axes, and by providing an explicit remediation comparison between temperature scaling and target\-domain fine\-tuning \(detailed in Section[1](https://arxiv.org/html/2608.26138#S1)\)\.

### 2\.3Calibration in Clinical Prediction Models

The clinical prediction literature requires external validation and calibration assessment for all prognostic models before deployment\[[7](https://arxiv.org/html/2608.26138#bib.bib7),[8](https://arxiv.org/html/2608.26138#bib.bib8)\]\. Calibration assessment — includingeceand reliability diagrams — is a standard component of reporting under TRIPOD guidelines\[[8](https://arxiv.org/html/2608.26138#bib.bib8)\]\. Guoet al\.\[[15](https://arxiv.org/html/2608.26138#bib.bib15)\]demonstrated that modern deep neural networks are systematically overconfident and that temperature scaling provides an effective post\-hoc calibration remedy\.

### 2\.4Algorithmic Fairness

Disparate impact and equalized odds are foundational fairness criteria in the machine learning fairness literature\[[16](https://arxiv.org/html/2608.26138#bib.bib16),[17](https://arxiv.org/html/2608.26138#bib.bib17)\]\. We extend these criteria to cross\-platform settings — where the group is defined by source platform rather than a demographic attribute — an application not systematically explored in mental healthnlp\. Prior fairness audits innlphave focused on demographic attributes\[[18](https://arxiv.org/html/2608.26138#bib.bib18)\]; platform\-as\-group fairness analysis is a natural extension for multi\-platform deployment scenarios, analogous to geographic or institutional group identifiers in healthcare equity literature\[[29](https://arxiv.org/html/2608.26138#bib.bib29),[30](https://arxiv.org/html/2608.26138#bib.bib30)\]\.

## 3Data and Preprocessing

### 3\.1Datasets

#### 3\.1\.1Training Platform \(Kaggle\)

A combined mental health dataset\[[19](https://arxiv.org/html/2608.26138#bib.bib19)\]aggregated from multiple social media platforms, consisting of text posts labelled for four categories:normal,depression,anxiety, andstress\. After preprocessing and stratified 70/15/15 splitting, the training set comprisedn=35,556n\{=\}35\{,\}556samples; the within\-platform test set comprisedn=7,620n\{=\}7\{,\}620samples\. The dataset is heavily skewed toward depression \(56\.6%56\.6\\%of the Kaggle partition, reflecting the training\-set distribution\), with anxiety \(7\.5%7\.5\\%\) and stress \(7\.2%7\.2\\%\) constituting small minorities\.

#### 3\.1\.2Cross\-Platform Test Set 1 \(Reddit / GoEmotions\)

The GoEmotions dataset\[[9](https://arxiv.org/html/2608.26138#bib.bib9)\]consists of58,00958\{,\}009English Reddit comments labelled with 27 fine\-grained emotion categories plus neutral\. The held\-out test partition \(n=6,257n\{=\}6\{,\}257after mapping\) was used\. Emotion labels were remapped to clinical categories as follows:sadness, grief, remorse, disappointment→\\todepression;nervousness, fear, anxiety→\\toanxiety;anger, annoyance, frustration→\\tostress; all remaining categories→\\tonormal\. These mappings approximate broad clinical syndromes from surface emotional expressions, following conventions established in prior mental healthnlpliterature\[[1](https://arxiv.org/html/2608.26138#bib.bib1),[5](https://arxiv.org/html/2608.26138#bib.bib5)\]\. These mappings are heuristic approximations: anger does not map cleanly to clinical stress in standard diagnostic taxonomy, and fear encompasses emotional states beyond anxiety\. Robustness to this mapping is assessed through four alternative mapping variants \(Section[4\.4](https://arxiv.org/html/2608.26138#S4.SS4)\)\.

#### 3\.1\.3Cross\-Platform Test Set 2 \(Twitter / dair\-ai/emotion\)

The dair\-ai/emotion dataset\[[10](https://arxiv.org/html/2608.26138#bib.bib10)\]consists of20,00020\{,\}000English tweets labelled with six emotion categories\. The test partition \(n=2,883n\{=\}2\{,\}883\) was used\. Remapping:sadness→\\todepression;fear→\\toanxiety;anger→\\tostress;joy, surprise, love→\\tonormal\. The Twitter distribution is more balanced:43\.6%43\.6\\%normal,30\.1%30\.1\\%depression,12\.3%12\.3\\%anxiety,14\.0%14\.0\\%stress\.

The full combined corpus spans112,211112\{,\}211samples across all three sources\. Label distribution shifts across platforms represent an independent confound in addition to domain shift \(Section[5\.5](https://arxiv.org/html/2608.26138#S5.SS5)\)\.

### 3\.2Preprocessing

All texts were tokenized using model\-specific tokenizers with maximum sequence length6464tokens\. Truncation analysis revealed that59\.2%59\.2\\%of Kaggle test\-set samples \(4,5084\{,\}508of7,6207\{,\}620\) exceed6464tokens \(mean word count118\.3118\.3; median67\.067\.0;p75p\_\{75\}:154\.0154\.0words\), compared with0\.0%0\.0\\%of Reddit samples \(mean13\.513\.5words\) and2\.8%2\.8\\%of Twitter samples \(mean19\.019\.0words\)\. This platform asymmetry — Kaggle text being66–9×9\\timeslonger than the cross\-platform texts — means models trained on truncated long\-form text learn signals from a compressed version of the clinical proxy narrative structure present in Kaggle posts, rather than from the shorter communicative forms characteristic of Reddit and Twitter\.

## 4Methodology

### 4\.1Models

Four transformer models spanning distinct pretraining and fine\-tuning regimes were evaluated \(Table[I](https://arxiv.org/html/2608.26138#S4.T1)\)\. All were fine\-tuned as four\-class sequence classifiers using AdamW\[[20](https://arxiv.org/html/2608.26138#bib.bib20)\]\(lr=2×10−5\\text\{lr\}\{=\}2\{\\times\}10^\{\-5\}, weight decay0\.010\.01\), linear warmup over10%10\\%of total steps, gradient norm clipping at1\.01\.0, and early stopping by maximum validation macro\-F1 over five epochs\. Maximum sequence length was6464tokens throughout\.

TABLE I:Transformer models evaluated in this study\.†GoEmotions\-RoBERTa Reddit results are non\-independent \(see Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.ModelParamsPretrainingBERT\[[21](https://arxiv.org/html/2608.26138#bib.bib21)\]110 MGeneral \(Wikipedia \+ Books\)RoBERTa\[[22](https://arxiv.org/html/2608.26138#bib.bib22)\]125 MGeneral \(dynamic masking\)Emotion\-DistilRoBERTa\[[23](https://arxiv.org/html/2608.26138#bib.bib23)\]82 MEmotion multi\-source corpusGoEmotions\-RoBERTa†125 MGoEmotions corpus \(Reddit\)BERT\(bert\-base\-uncased\[[21](https://arxiv.org/html/2608.26138#bib.bib21)\]\):110110M parameters; general\-domain pretraining on English Wikipedia and BookCorpus; WordPiece tokenizer \(30,52230\{,\}522token vocabulary\)\.

RoBERTa\(roberta\-base\[[22](https://arxiv.org/html/2608.26138#bib.bib22)\]\):125125M parameters; general\-domain pretraining with dynamic masking and larger mini\-batches; BPE tokenizer \(50,26550\{,\}265token vocabulary\)\.

Emotion\-DistilRoBERTa\(j\-hartmann/emotion\-english\-distilroberta\-base\[[23](https://arxiv.org/html/2608.26138#bib.bib23)\]\):8282M parameters; DistilRoBERTa fine\-tuned on a multi\-source emotion classification corpus that includes GoEmotions as a component\. This model has partial pretraining overlap with the Reddit evaluation domain\.

GoEmotions\-RoBERTa\(SamLowe/roberta\-base\-go\_emotions\):125125M parameters; RoBERTa\-base fine\-tuned on the full GoEmotions corpus — the same source as the Reddit test set\. Its Reddit evaluation results are therefore non\-independent: the model encountered examples from the test\-set distribution during training\. These results are reported for reference alongside the three independently evaluated models, but should be interpreted as an in\-distribution performance ceiling, not as evidence of generalisable cross\-platform robustness\.

### 4\.2Evaluation Metrics

#### 4\.2\.1Discriminative Performance

Macro\-averaged one\-vs\-restauc, macro F1, and accuracy were computed on each test set\. Per\-classaucvalues with95%95\\%confidence intervals were computed using the method of DeLonget al\.\[[24](https://arxiv.org/html/2608.26138#bib.bib24)\]\.

#### 4\.2\.2Calibration

Expected Calibration Error \(ece\) withM=10M\{=\}10equal\-width bins:

ece=∑m=1M\|Bm\|n​\|acc⁡\(Bm\)−conf⁡\(Bm\)\|,\\textsc\{ece\}=\\sum\_\{m=1\}^\{M\}\\frac\{\|B\_\{m\}\|\}\{n\}\\bigl\|\\operatorname\{acc\}\(B\_\{m\}\)\-\\operatorname\{conf\}\(B\_\{m\}\)\\bigr\|,\(1\)whereBmB\_\{m\}is the set of samples in binmm,conf⁡\(Bm\)\\operatorname\{conf\}\(B\_\{m\}\)is the mean predicted probability, andacc⁡\(Bm\)\\operatorname\{acc\}\(B\_\{m\}\)is the classification accuracy within the bin\. Bootstrap95%95\\%CIs were computed withB=1,000B\{=\}1\{,\}000resamples\.

#### 4\.2\.3Statistical Significance

Pairwise macroauccomparisons used bootstrap standard errors \(B=2,000B\{=\}2\{,\}000\) with Bonferroni correction applied within each model’s family of three pairwise platform comparisons \(α′=0\.05/3=0\.0167\\alpha^\{\\prime\}\{=\}0\.05/3\{=\}0\.0167\); each model’s comparisons are treated as a separate family given architectural independence across models\.

#### 4\.2\.4Disparate Impact \(DI\)

Symmetricdifor classcc:

dic=min⁡\(P​\(y^=c∣G=A\)P​\(y^=c∣G=B\),P​\(y^=c∣G=B\)P​\(y^=c∣G=A\)\),\\textsc\{di\}\_\{c\}=\\min\\\!\\left\(\\frac\{P\(\\hat\{y\}\{=\}c\\mid G\{=\}A\)\}\{P\(\\hat\{y\}\{=\}c\\mid G\{=\}B\)\},\\,\\frac\{P\(\\hat\{y\}\{=\}c\\mid G\{=\}B\)\}\{P\(\\hat\{y\}\{=\}c\\mid G\{=\}A\)\}\\right\),\(2\)whereG∈\{Kaggle,target platform\}G\\in\\\{\\text\{Kaggle\},\\,\\text\{target platform\}\\\}\.di∈\(0,1\]\\textsc\{di\}\\in\(0,1\];di<0\.80\\textsc\{di\}\{<\}0\.80violates the four\-fifths rule\[[16](https://arxiv.org/html/2608.26138#bib.bib16)\];di<0\.50\\textsc\{di\}\{<\}0\.50constitutes a severe disparity\.

#### 4\.2\.5Equalized Odds Difference \(EOD\)

eodc=\|tprc​\(Kaggle\)−tprc​\(target\)\|,\\textsc\{eod\}\_\{c\}=\\bigl\|\\textsc\{tpr\}\_\{c\}\(\\text\{Kaggle\}\)\-\\textsc\{tpr\}\_\{c\}\(\\text\{target\}\)\\bigr\|,\(3\)wheretprc=P\(y^=c∣y=c,G\)\\textsc\{tpr\}\_\{c\}=P\(\\hat\{y\}\{=\}c\\mid y\{=\}c,\\,G\)\.eod=0\\textsc\{eod\}\{=\}0indicates perfect equalized odds\.

#### 4\.2\.6Attribution Stability

Token importance was estimated by gradient\-based saliency\[[25](https://arxiv.org/html/2608.26138#bib.bib25)\], implemented via the Captum library\[[33](https://arxiv.org/html/2608.26138#bib.bib33)\]:

si=‖∂P​\(y∣𝐱\)∂Ei‖2,s\_\{i\}=\\left\\\|\\frac\{\\partial P\(y\\mid\\mathbf\{x\}\)\}\{\\partial E\_\{i\}\}\\right\\\|\_\{2\},\(4\)whereEiE\_\{i\}is the embedding of tokenii\. Per\-token scores were aggregated by word type across a platform sample \(n=200n\{=\}200per platform\)\. Jaccard similarity between top\-KKword sets from two platforms:

JK​\(A,B\)=\|topK⁡\(A\)∩topK⁡\(B\)\|\|topK⁡\(A\)∪topK⁡\(B\)\|\.J\_\{K\}\(A,B\)=\\frac\{\|\\operatorname\{top\}\_\{K\}\(A\)\\cap\\operatorname\{top\}\_\{K\}\(B\)\|\}\{\|\\operatorname\{top\}\_\{K\}\(A\)\\cup\\operatorname\{top\}\_\{K\}\(B\)\|\}\.\(5\)Reported atK∈\{5,10,15,20\}K\\in\\\{5,10,15,20\\\}\.

### 4\.3Temperature Scaling

Post\-hoc temperature scaling\[[15](https://arxiv.org/html/2608.26138#bib.bib15)\]recalibrates predicted probabilities via:

pT=softmax⁡\(𝐳/T\),p\_\{T\}=\\operatorname\{softmax\}\(\\mathbf\{z\}/T\),\(6\)where𝐳\\mathbf\{z\}is the raw pre\-softmax logit vector and scalarT\>0T\{\>\}0\. TemperatureT∗T^\{\*\}minimises negative log\-likelihood on a held\-out calibration split \(10%10\\%of the target platform test set, stratified by label\)\. Raw pre\-softmax logits were saved at inference time and used directly for temperature scaling\.

### 4\.4Sensitivity Analysis

Four label mapping schemes were evaluated to assess robustness to the emotion→\\toclinical category remapping:

- •A: Original 4\-class \(normal / depression / anxiety / stress\);
- •B: Binary \(normal vs\. any mental health condition\);
- •C: 3\-class \(normal / depression / distress\);
- •D: Distress superclass \(anxiety and stress merged into a single distress category\)\.

No model retraining was required; ground\-truth labels and predicted probabilities from the primary analysis were remapped\.

### 4\.5Multi\-Seed Training Protocol

To characterise training variance, all four models were trained with five independent random seeds \(\{42,0,1,7,123\}\\\{42,0,1,7,123\\\}\)\. Seeds control all stochastic elements: parameter initialisation, DataLoader shuffling, and dropout mask sampling, with all relevant random\-number generators \(Python, NumPy, PyTorch CPU and GPU, and Hugging Face Transformers\) seeded consistently\. All results in Tables[II](https://arxiv.org/html/2608.26138#S5.T2)–[X](https://arxiv.org/html/2608.26138#S5.T10)report mean±\\pmSD across the five seeds\. Within\-platform performance variance was small \(aucSD≤0\.001\\leq 0\.001for all models\), confirming that theseed=42results are representative\.

### 4\.6Fine\-Tuning Baseline

To contextualise temperature scaling, we evaluated direct target\-domain fine\-tuning as an alternative use of the same labelled target\-platform examples\. Using the same10%10\\%stratified calibration split employed in Section[4\.3](https://arxiv.org/html/2608.26138#S4.SS3)\(n=625n\{=\}625for Reddit;n=286n\{=\}286for Twitter\), each model was fine\-tuned for three additional epochs using AdamW \(lr=2×10−5\\text\{lr\}\{=\}2\{\\times\}10^\{\-5\}, weight decay0\.010\.01,10%10\\%linear warmup\) and evaluated on the remaining90%90\\%of the target platform test set\. This comparison evaluates, under matched data\-volume conditions, whether labelled target\-platform examples yield greater benefit as a calibration signal \(temperature scaling\) or as an additional training signal \(fine\-tuning\), noting that the two approaches extract qualitatively different information from the same split\.

## 5Results

Results are presented sequentially across each of the fivecpfeaxes: discriminative performance \(Axis 1\) establishes the primary failure; statistical significance \(Axis 3\) confirms it is not a sampling artefact; prediction equity \(Axis 4\) characterises distributional consequences; calibration \(Axis 2\) evaluates whether post\-hoc correction helps; and attribution stability \(Axis 5\) probes mechanism\. Thecpfeframework is illustrated in Fig\.[1](https://arxiv.org/html/2608.26138#S5.F1)\.

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig01_cpfe_framework.png)Figure 1:The Cross\-Platform Fairness Evaluation \(cpfe\) framework\. Five evaluation axes are applied to models trained on Kaggle and evaluated on Reddit and Twitter\.### 5\.1Within\-Platform Performance

All four models achieved strong within\-platform performance on the Kaggle test set \(Table[II](https://arxiv.org/html/2608.26138#S5.T2)\)\. RoBERTa achieved the highest macroauc\(0\.9870\.987\) and macro F1 \(0\.8830\.883\); GoEmotions\-RoBERTa was closely comparable \(auc0\.9850\.985, F10\.8710\.871\)\. Within\-platformecewas0\.0560\.056–0\.0600\.060, indicating good calibration on the training distribution\. The stress class showed the lowest per\-class F1 across all models \(per\-class values available in the supplementary code repository\), attributable to its small support \(n=549n\{=\}549,7\.2%7\.2\\%\) and conceptual proximity to anxiety\.

TABLE II:Within\-platform performance on the Kaggle mental health corpus test set \(n=7,620n\{=\}7\{,\}620\) for four\-class clinical proxy label classification\. Values are means±\\pmSD across five training seeds\.eceis reported with95%95\\%bootstrap CIs \(B=1,000B\{=\}1\{,\}000bootstrap samples, stratified\)\.ModelAccuracyF1\-macroF1\-weightedAUCECE \[95% CI\]BERT0\.9300\.8740\.9310\.9840\.060 \[0\.055, 0\.066\]RoBERTa0\.9360\.8830\.9370\.9870\.056 \[0\.052, 0\.062\]Emotion\-DistilRoBERTa0\.9280\.8620\.9280\.9830\.058 \[0\.053, 0\.064\]GoEmotions\-RoBERTa0\.9300\.8710\.9310\.9850\.059 \[0\.054, 0\.064\]
### 5\.2Cross\-Platform AUC Degradation \(Axis 1\)

TABLE III:Cross\-platform macroauc, F1\-macro, andecefor Reddit GoEmotions \(n=6,257n\{=\}6\{,\}257\) and Twitter dair\-ai/emotion \(n=2,883n\{=\}2\{,\}883\) test sets\.Δ\\Deltaauc% is computed relative to within\-platform Kaggleauc\(Table[II](https://arxiv.org/html/2608.26138#S5.T2)\)\. Values are means across five training seeds\.†GoEmotions\-RoBERTa was fine\-tuned on the GoEmotions corpus, which constitutes the Reddit evaluation set; its Reddit results are non\-independent \(Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.ModelPlatformAccuracyF1\-macroAUCECE \[95% CI\]Δ\\DeltaAUC%BERTReddit0\.7540\.3030\.6450\.229 \[0\.218, 0\.239\]−34\.5%\-34\.5\\%RoBERTaReddit0\.7680\.3040\.6370\.221 \[0\.210, 0\.231\]−35\.4%\-35\.4\\%Emotion\-DistilRoBERTaReddit0\.7680\.3320\.6850\.208 \[0\.198, 0\.218\]−30\.3%\-30\.3\\%GoEmotions\-RoBERTa†Reddit0\.7860\.3180\.7030\.196 \[0\.186, 0\.205\]−28\.6%\-28\.6\\%BERTTwitter0\.4600\.3170\.5960\.506 \[0\.489, 0\.524\]−39\.5%\-39\.5\\%RoBERTaTwitter0\.4600\.2840\.6030\.514 \[0\.497, 0\.533\]−38\.9%\-38\.9\\%Emotion\-DistilRoBERTaTwitter0\.4090\.2900\.6110\.542 \[0\.524, 0\.560\]−37\.9%\-37\.9\\%GoEmotions\-RoBERTa†Twitter0\.4660\.3060\.6050\.499 \[0\.481, 0\.517\]−38\.6%\-38\.6\\%All three independently evaluated models show systematicaucdegradation of30\.330\.3–35\.4%35\.4\\%on Reddit and37\.937\.9–39\.5%39\.5\\%on Twitter \(Table[III](https://arxiv.org/html/2608.26138#S5.T3); Fig\.[2](https://arxiv.org/html/2608.26138#S5.F2)\)\. Macroaucfalls from0\.9830\.983–0\.9870\.987within\-platform to0\.6370\.637–0\.6850\.685on Reddit and0\.5960\.596–0\.6110\.611on Twitter\. F1\-macro collapses cross\-platform to0\.3030\.303–0\.3320\.332on Reddit — a6363–68%68\\%reduction from in\-domain performance \(0\.8620\.862–0\.8830\.883\) — indicating that even a basic recall\-precision balance across the four proxy classes is unachievable on out\-of\-domain data\.

A trivial majority\-class classifier \(always predicting ‘normal’\) achieves F1\-macro≈0\.224\{\\approx\}0\.224on Reddit and≈0\.266\{\\approx\}0\.266on Twitter\. The models’ cross\-platform F1\-macro of0\.3030\.303–0\.3320\.332on Reddit represents marginal but consistent improvement over this degenerate baseline, confirming that performance remains above the trivial majority\-class baseline despite the substantial degradation\.

GoEmotions\-RoBERTa shows the smallest Redditaucdrop \(28\.6%28\.6\\%,auc0\.7030\.703\), consistent with its pretraining on the GoEmotions corpus — but this result is non\-independent and represents an in\-distribution ceiling rather than a cross\-platform benchmark \(†; Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\. On Twitter, all four models converge toauc0\.5960\.596–0\.6110\.611\(drops37\.937\.9–39\.5%39\.5\\%\), confirming that the GoEmotions pretraining advantage does not extend to the Twitter distribution\.

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig02_auc_degradation.jpg)Figure 2:All three independently evaluated models showaucdrops exceeding30%30\\%cross\-platform\. Error bars show standard deviation across five random seeds\.
### 5\.3Per\-Class AUC with DeLong Confidence Intervals

Table[IV](https://arxiv.org/html/2608.26138#S5.T4)reports per\-class one\-vs\-restaucwith DeLong confidence intervals\. The stress classaucon Twitter is only marginally above chance for all models \(0\.5220\.522–0\.5420\.542; see also Fig\.[3](https://arxiv.org/html/2608.26138#S5.F3)\)\. RoBERTa stressauc=0\.522=0\.522\[95% CI:0\.4920\.492,0\.5510\.551\] has its lower bound below0\.5000\.500, meaning this model cannot reliably distinguish posts mapped to the stress proxy class on Twitter from non\-stress posts at above\-chance level\. Depression proxy\-classaucon Reddit \(0\.5970\.597–0\.6280\.628\) falls below commonly cited clinical discrimination benchmarks\[[31](https://arxiv.org/html/2608.26138#bib.bib31),[32](https://arxiv.org/html/2608.26138#bib.bib32)\]\(discussed further in Section[6](https://arxiv.org/html/2608.26138#S6)\)\. Anxietyaucon Reddit is somewhat better preserved \(0\.6860\.686–0\.7050\.705\), possibly reflecting partial coherence between the GoEmotions fear/nervousness/anxiety labels and the clinical anxiety construct\.

TABLE IV:Per\-class one\-vs\-restaucwith 95% DeLong confidence intervals across three platforms\. Values are means across five training seeds\.†Non\-independent Reddit evaluation \(Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.ModelPlatformNormalDepressionAnxietyStressAUC \[95% CI\]AUC \[95% CI\]AUC \[95% CI\]AUC \[95% CI\]Panel A: Kaggle \(within\-platform\)BERTKaggle0\.991 \[0\.989, 0\.993\]0\.988 \[0\.986, 0\.990\]0\.988 \[0\.984, 0\.992\]0\.969 \[0\.962, 0\.976\]RoBERTaKaggle0\.993 \[0\.992, 0\.995\]0\.990 \[0\.989, 0\.992\]0\.991 \[0\.987, 0\.994\]0\.972 \[0\.966, 0\.978\]Emotion\-DistilRoBERTaKaggle0\.993 \[0\.992, 0\.995\]0\.988 \[0\.986, 0\.990\]0\.986 \[0\.981, 0\.991\]0\.966 \[0\.959, 0\.972\]GoEmotions\-RoBERTaKaggle0\.992 \[0\.990, 0\.994\]0\.989 \[0\.987, 0\.991\]0\.988 \[0\.984, 0\.992\]0\.971 \[0\.966, 0\.976\]Panel B: Reddit \(cross\-platform\)BERTReddit0\.651 \[0\.634, 0\.669\]0\.597 \[0\.568, 0\.626\]0\.693 \[0\.647, 0\.739\]0\.638 \[0\.614, 0\.662\]RoBERTaReddit0\.648 \[0\.631, 0\.666\]0\.599 \[0\.571, 0\.627\]0\.686 \[0\.636, 0\.736\]0\.616 \[0\.591, 0\.640\]Emotion\-DistilRoBERTaReddit0\.694 \[0\.677, 0\.712\]0\.612 \[0\.582, 0\.641\]0\.705 \[0\.653, 0\.757\]0\.730 \[0\.708, 0\.752\]GoEmotions\-RoBERTa†Reddit0\.726 \[0\.710, 0\.744\]0\.628 \[0\.600, 0\.656\]0\.702 \[0\.649, 0\.756\]0\.756 \[0\.734, 0\.777\]Panel C: Twitter \(cross\-platform\)BERTTwitter0\.622 \[0\.602, 0\.642\]0\.603 \[0\.580, 0\.625\]0\.628 \[0\.596, 0\.661\]0\.530 \[0\.500, 0\.560\]RoBERTaTwitter0\.635 \[0\.615, 0\.655\]0\.607 \[0\.584, 0\.629\]0\.649 \[0\.617, 0\.680\]0\.522 \[0\.492, 0\.551\]Emotion\-DistilRoBERTaTwitter0\.608 \[0\.587, 0\.628\]0\.586 \[0\.562, 0\.609\]0\.714 \[0\.682, 0\.747\]0\.535 \[0\.503, 0\.567\]GoEmotions\-RoBERTa†Twitter0\.642 \[0\.622, 0\.662\]0\.622 \[0\.599, 0\.643\]0\.614 \[0\.579, 0\.649\]0\.542 \[0\.512, 0\.571\]![Refer to caption](https://arxiv.org/html/2608.26138v1/fig03_f1_heatmap.jpg)Figure 3:Per\-class F1 score heatmap for all models and platforms\.
### 5\.4Statistical Significance of Performance Gaps \(Axis 3\)

All1212pairwiseauccomparisons reach significance under Bonferroni\-corrected bootstrapZZ\-tests \(cpfeAxis 3; Table[V](https://arxiv.org/html/2608.26138#S5.T5)\)\. Kaggle\-vs\-Twitter comparisons produce the largestZZ\-statistics \(47\.747\.7–51\.951\.9\), reflecting the greater absoluteaucdifference for this platform pair\. Reddit\-vs\-Twitter comparisons are also significant for all four models \(Z=2\.89Z\{=\}2\.89–8\.258\.25\), indicating that Reddit and Twitter represent statistically distinguishable degradation regimes, though the practical magnitude of the Reddit–Twitter difference is smaller than the within\-to\-cross\-platform gap\.

TABLE V:Pairwise macroauccomparisons using bootstrap standard errors under Bonferroni correction \(33pairwise comparisons per model;α′=0\.0167\\alpha^\{\\prime\}\{=\}0\.0167\)\. All1212comparisons are significant\. Significance:p∗∗<0\.01\{\}^\{\*\*\}p\{<\}0\.01;p∗⁣∗∗<0\.001\{\}^\{\*\*\*\}p\{<\}0\.001\(Bonferroni\-corrected threshold\)\.†Non\-independent Reddit evaluation \(Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.ModelComparisonAUC1AUC2𝚫\\bm\{\\Delta\}AUC𝒁\\bm\{Z\}𝒑\\bm\{p\}Sig\.?BERTKaggle vs Reddit0\.9840\.6450\.33937\.6<0\.001\{<\}0\.001Yes∗∗∗BERTKaggle vs Twitter0\.9840\.5960\.38851\.9<0\.001\{<\}0\.001Yes∗∗∗BERTReddit vs Twitter0\.6450\.5960\.0494\.230\.000023Yes∗∗∗RoBERTaKaggle vs Reddit0\.9870\.6370\.34938\.0<0\.001\{<\}0\.001Yes∗∗∗RoBERTaKaggle vs Twitter0\.9870\.6030\.38449\.4<0\.001\{<\}0\.001Yes∗∗∗RoBERTaReddit vs Twitter0\.6370\.6030\.0342\.890\.004Yes∗∗Emotion\-DistilRoBERTaKaggle vs Reddit0\.9830\.6850\.29831\.6<0\.001\{<\}0\.001Yes∗∗∗Emotion\-DistilRoBERTaKaggle vs Twitter0\.9830\.6110\.37247\.7<0\.001\{<\}0\.001Yes∗∗∗Emotion\-DistilRoBERTaReddit vs Twitter0\.6850\.6110\.0756\.16<0\.001\{<\}0\.001Yes∗∗∗GoEmotions\-RoBERTa†Kaggle vs Reddit0\.9850\.7030\.28230\.3<0\.001\{<\}0\.001Yes∗∗∗GoEmotions\-RoBERTa†Kaggle vs Twitter0\.9850\.6050\.38050\.0<0\.001\{<\}0\.001Yes∗∗∗GoEmotions\-RoBERTa†Reddit vs Twitter0\.7030\.6050\.0988\.25<0\.001\{<\}0\.001Yes∗∗∗
### 5\.5Disparate Impact and Equalized Odds \(Axis 4\)

TABLE VI:Symmetric Disparate Impact \(di\) and Equalized Odds Difference \(eod\) across platforms, treating source platform \(Kaggle vs\. target\) as the group variable\.di<0\.80\{<\}0\.80violates the four\-fifths rule\[[16](https://arxiv.org/html/2608.26138#bib.bib16)\];di<0\.50\{<\}0\.50constitutes a severe disparity \(bold\)\. Values are means across five training seeds\.†Non\-independent Reddit evaluation \(Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.ModelPlatformDI NormalDI Dep\.DI Anx\.DI StressEODmax\(class\)BERTReddit0\.3180\.1670\.1060\.1310\.830 \(depression\)RoBERTaReddit0\.3120\.1330\.1090\.0980\.830 \(depression\)Emotion\-DistilRoBERTaReddit0\.3160\.1560\.1490\.1440\.790 \(depression\)GoEmotions\-RoBERTa†Reddit0\.3040\.1290\.0990\.0910\.802 \(depression\)BERTTwitter0\.5050\.6960\.2310\.4010\.787 \(anxiety\)RoBERTaTwitter0\.5240\.7900\.1060\.0540\.831 \(anxiety\)Emotion\-DistilRoBERTaTwitter0\.7960\.9090\.2030\.0690\.753 \(anxiety\)GoEmotions\-RoBERTa†Twitter0\.4540\.6420\.2100\.1380\.795 \(anxiety\)divalues for the three mental health proxy classes \(depression, anxiety, stress\) on Reddit range from0\.0910\.091to0\.1670\.167\(cpfeAxis 4; Table[VI](https://arxiv.org/html/2608.26138#S5.T6); Fig\.[4](https://arxiv.org/html/2608.26138#S5.F4)\) — substantially below the0\.500\.50severe\-disparity threshold and the0\.800\.80four\-fifths rule\[[16](https://arxiv.org/html/2608.26138#bib.bib16)\]threshold; the Normal class showsdi0\.3040\.304–0\.3180\.318, also substantially below both fairness thresholds\. At the level of conditional error rates, maximumeodfor depression on Reddit reaches0\.8300\.830\(BERT and RoBERTa\), meaning the depression proxy\-class true positive rate is8383percentage points lower on Reddit than on Kaggle\.

The reporteddivalues reflect a combination of genuine prediction rate disparities and platform\-level label distribution differences\. Prior\-shift\-adjusteddicalculations — computed by re\-weighting predictions to the reference\-platform class prevalences before applying Equation[2](https://arxiv.org/html/2608.26138#S4.E2)— confirm that a substantial disparity persists beyond what prevalence differences alone explain \(adjusteddi:0\.110\.11–0\.290\.29across mental health proxy classes on Reddit; see Supplementary Table S1\)\. Theeodvalues, which condition on true class membership, are less susceptible to this confound and remain severely elevated\.

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig04_di_heatmap.jpg)Figure 4:Alldivalues fall substantially below the0\.800\.80four\-fifths threshold across both platforms\. Symmetric Disparate Impact \(di\) heatmap for all four models on Reddit \(top\) and Twitter \(bottom\)\. Green \(≥0\.80\\geq 0\.80\): four\-fifths rule compliance; yellow/red: violation\. Reference platform: Kaggle\.
### 5\.6Temperature Scaling Recalibration \(Axis 2\)

TABLE VII:Platform\-specific temperature scaling results\. Optimal temperatureT∗T^\{\*\}was fitted on a10%10\\%stratified calibration split drawn from the target\-platform test set;ecereported on the remaining90%90\\%\. Post\-calibrationecevalues may be slightly optimistic relative to a fully held\-out deployment sample \(see Section[7](https://arxiv.org/html/2608.26138#S7)\)\.aucis invariant to temperature scaling by construction\. Values are means across five training seeds\.ModelPlatformT∗T^\{\*\}ECE BeforeECE AfterECE ReductionΔ\\DeltaAUCBERTReddit4\.310\.2250\.05774\.5%−0\.004\-0\.004BERTTwitter8\.040\.5080\.02794\.7%\+0\.014\+0\.014RoBERTaReddit3\.880\.2210\.07068\.2%−0\.010\-0\.010RoBERTaTwitter9\.520\.5090\.01896\.4%\+0\.004\+0\.004Emotion\-DistilRoBERTaReddit3\.160\.2100\.03384\.3%\+0\.002\+0\.002Emotion\-DistilRoBERTaTwitter8\.270\.5410\.04392\.0%\+0\.033\+0\.033GoEmotions\-RoBERTaReddit3\.140\.1940\.06666\.3%−0\.004\-0\.004GoEmotions\-RoBERTaTwitter6\.810\.5030\.03992\.2%\+0\.002\+0\.002Mean across all model–platform pairs0\.04488\.0%<\|0\.01\|<\|0\.01\|Temperature scaling reduceseceacross all conditions \(Table[VII](https://arxiv.org/html/2608.26138#S5.T7); Fig\.[5](https://arxiv.org/html/2608.26138#S5.F5)\)\. Optimal temperatures range fromT∗=3\.14T^\{\*\}\{=\}3\.14\(GoEmotions\-RoBERTa on Reddit\) toT∗=9\.52T^\{\*\}\{=\}9\.52\(RoBERTa on Twitter\), reflecting the degree of overconfidence\. Meaneceacross all models and platforms falls from0\.3640\.364before recalibration to0\.0440\.044after, a reduction of88\.0%88\.0\\%\(range:66\.366\.3–96\.4%96\.4\\%\)\.

Macroaucis invariant to temperature scaling \(mean\|Δ​auc\|=0\.009\|\\Delta\\text\{\{auc\}\{\}\}\|\{=\}0\.009, range−0\.010\-0\.010to\+0\.033\+0\.033\), consistent with the mathematical property thatsoftmax⁡\(𝐳/T\)\\operatorname\{softmax\}\(\\mathbf\{z\}/T\)is a monotonic transformation of model scores that cannot alter prediction ranking\. This confirms that recalibration cannot address the discriminative deficit observed cross\-platform: a model withauc0\.5960\.596–0\.6110\.611on Twitter retains only marginal discriminative utility relative to commonly cited clinical benchmarks, and post\-hoc recalibration alone cannot address this discriminative deficit\.

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig05_reliability_diagrams.jpg)Figure 5:Calibration degrades severely cross\-platform; within\-platform curves lie near the diagonal while cross\-platform curves deviate substantially\. Reliability diagrams for all four models across three platforms; each panel shows mean predicted confidence against observed accuracy inM=10M\{=\}10equal\-width bins\.
### 5\.7Attribution Stability \(Axis 5\)

AtK=10K\{=\}10,1414of1616model\-class pairs showJ=0\.000J\{=\}0\.000for Kaggle\-to\-Twitter comparisons \(cpfeAxis 5; Table[VIII](https://arxiv.org/html/2608.26138#S5.T8); Fig\.[9](https://arxiv.org/html/2608.26138#S5.F9)\), indicating that the top\-10 predictive tokens for a given class on Kaggle share zero members with the top\-10 tokens on Twitter\. Two model–class pairs show non\-zero Jaccard similarity for this comparison: RoBERTa\-normal \(J=0\.111J\{=\}0\.111\) and Emotion\-DistilRoBERTa\-normal \(J=0\.111J\{=\}0\.111\), both involving the normal class, consistent with its higher base\-rate prevalence providing more stable attribution signal\. For Kaggle\-to\-Reddit comparisons,88of1616pairs showJ=0\.000J\{=\}0\.000, with non\-zero values ranging from0\.0530\.053to0\.1760\.176\.

TABLE VIII:Cross\-platform Jaccard similarity \(JJ\) of top\-K=10K\{=\}10gradient\-saliency attribution token sets for each of1616model\-class pairs \(44models×\\times44classes\)\.J=0J\{=\}0denotes complete token vocabulary divergence atK=10K\{=\}10;J=1J\{=\}1denotes complete identity\. Random baseline:J≈0\.0001J\{\\approx\}0\.0001for a50,26550\{,\}265\-token vocabulary\.†Non\-independent Reddit evaluation \(Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.ModelClassJJ\(Kaggle→\\toReddit\)JJ\(Kaggle→\\toTwitter\)JJ\(Reddit→\\toTwitter\)BERTNormal0\.0000\.0000\.053BERTDepression0\.0530\.0000\.053BERTAnxiety0\.0000\.0000\.053BERTStress0\.0000\.0000\.111RoBERTaNormal0\.0530\.1110\.111RoBERTaDepression0\.0000\.0000\.053RoBERTaAnxiety0\.1760\.0000\.111RoBERTaStress0\.0000\.0000\.111Emotion\-DistilRoBERTaNormal0\.0530\.1110\.053Emotion\-DistilRoBERTaDepression0\.0530\.0000\.000Emotion\-DistilRoBERTaAnxiety0\.0000\.0000\.000Emotion\-DistilRoBERTaStress0\.0530\.0000\.053GoEmotions\-RoBERTa†Normal0\.0530\.0000\.111GoEmotions\-RoBERTa†Depression0\.0000\.0000\.111GoEmotions\-RoBERTa†Anxiety0\.0000\.0000\.111GoEmotions\-RoBERTa†Stress0\.0530\.0000\.053Proxy\-clinical signal retention analysis — using a reference vocabulary of4545terms drawn from DSM\-5 symptom descriptions and established clinical lexicons — shows that top\-10 model features match this vocabulary at rates of0–30%30\\%on Kaggle but uniformly at0%0\\%on Reddit and Twitter\. On Kaggle, model attention concentrates on proxy\-clinical vocabulary consistent with DSM\-adjacent terminology\. On Reddit and Twitter, attribution patterns shift predominantly toward platform\-specific conversational vocabulary \(see Figs\.[6](https://arxiv.org/html/2608.26138#S5.F6)–[8](https://arxiv.org/html/2608.26138#S5.F8)\)\. \(Gradient saliency has known limitations; near\-zero Jaccard values should be interpreted as consistent with attribution instability rather than as definitive attribution collapse — see Section[7](https://arxiv.org/html/2608.26138#S7), item 3\.\)

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig06_attr_goemotions.jpg)Figure 6:Top\-15 gradient saliency token attributions for GoEmotions\-RoBERTa, Depression class \(top\) and Stress class \(bottom\), across Kaggle \(blue\), Reddit \(green\), and Twitter \(orange\)\. Token scores are normalised per platform\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig07_attr_bert.jpg)Figure 7:Top\-15 gradient saliency token attributions for BERT, Depression class, across Kaggle, Reddit, and Twitter\. Token scores are normalised per platform\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig08_attr_roberta.jpg)Figure 8:Top\-15 gradient saliency token attributions for RoBERTa, Depression class, across Kaggle, Reddit, and Twitter\. Token scores are normalised per platform\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig09_jaccard_heatmap.jpg)Figure 9:Jaccard similarity heatmap for top\-K=10K\{=\}10attribution token sets across all four models and four proxy classes\.
### 5\.8Fine\-Tuning vs\. Temperature Scaling

TABLE IX:Target\-domain fine\-tuning vs\. temperature scaling comparison\. Each model was fine\-tuned for three additional epochs on the same10%10\\%stratified calibration split\.aucandecereported on the remaining90%90\\%of the target\-platform test set\.Δ\\DeltaAUC==fine\-tuned AUC−\-baseline cross\-platform AUC\. MeanΔ\\DeltaAUC across all model–platform pairs:\+0\.216\+0\.216\. Fine\-tuning conducted under single seed \(seed = 42\)\. Base AUC values are single\-seed \(seed=\{=\}42\) results for comparability with fine\-tuned AUC; five\-seed mean AUC values for the same models appear in Table[III](https://arxiv.org/html/2608.26138#S5.T3)\.†Non\-independent Reddit evaluation \(Section[4\.1](https://arxiv.org/html/2608.26138#S4.SS1)\)\.‡Emotion\-DistilRoBERTa Twitter fine\-tuned AUC of0\.9610\.961should be interpreted with caution given the small sample \(ncal=286n\_\{\\rm cal\}\{=\}286\) and single\-seed design\.ModelPlatformncaln\_\{\\rm cal\}nevaln\_\{\\rm eval\}Base AUCFT AUC𝚫\\bm\{\\Delta\}AUCFT ECETS ECEBERTReddit6255,6320\.6420\.819\+0\.177\+0\.1770\.0990\.047BERTTwitter2862,5970\.5980\.770\+0\.172\+0\.1720\.0740\.033RoBERTaReddit6255,6320\.6340\.862\+0\.228\+0\.2280\.0820\.037RoBERTaTwitter2862,5970\.6060\.805\+0\.198\+0\.1980\.0610\.007Emotion\-DistilRoBERTaReddit6255,6320\.6810\.846\+0\.166\+0\.1660\.0820\.033Emotion\-DistilRoBERTa‡Twitter2862,5970\.6110\.961\+0\.350\+0\.3500\.0190\.043GoEmotions\-RoBERTa†Reddit6255,6320\.7010\.925\+0\.224\+0\.2240\.0460\.038GoEmotions\-RoBERTa†Twitter2862,5970\.6080\.822\+0\.215\+0\.2150\.0940\.037Mean\+0\.216\\bm\{\+0\.216\}In this single\-seed fine\-tuning experiment, meanaucimproved by0\.2160\.216across model–platform pairs \(Table[IX](https://arxiv.org/html/2608.26138#S5.T9); Fig\.[10](https://arxiv.org/html/2608.26138#S5.F10)\)\. The Emotion\-DistilRoBERTa Twitter result \(Δ\\Deltaauc=\+0\.350=\{\+\}0\.350;ncal=286n\_\{\\rm cal\}\{=\}286\) inflates this mean; excluding that pair gives a mean gain of0\.1970\.197\. These results should be treated as indicative rather than definitive given the single\-seed design\. Where labelled target\-platform samples are available, fine\-tuning recovers discriminative performance substantially more effectively than post\-hoc recalibration\. Temperature scaling remains appropriate when fine\-tuning is computationally infeasible or when only calibration correction — rather than discriminative recovery — is the operational goal\.

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig10_ft_vs_ts.jpg)Figure 10:Temperature scaling vs\. target\-domain fine\-tuning as calibration remediation strategies for all four models on Reddit and Twitter\.
### 5\.9Sensitivity Analysis

Cross\-platform degradation patterns are consistent under all four alternative label mapping schemes \(Table[X](https://arxiv.org/html/2608.26138#S5.T10); Fig\.[11](https://arxiv.org/html/2608.26138#S5.F11)\)\. Under Mapping B \(binary\), which eliminates all fine\-grained class\-specific label ambiguity, Reddit drops remain26\.826\.8–34\.7%34\.7\\%and Twitter drops remain35\.335\.3–39\.0%39\.0\\%\. The consistency of cross\-platformaucdegradation across all four alternative mapping schemes — including Mapping D, which collapses anxiety and stress into a single distress superclass — provides the primary empirical basis for concluding that generalisation failure is not an artefact of the label\-mapping heuristic\.

TABLE X:Cross\-platformaucdegradation under four label mapping schemes: \(A\) 4\-class primary; \(B\) binary \(normal vs\. any mental health condition\); \(C\) 3\-class \(normal/depression/distress\); \(D\) distress superclass \(collapsing anxiety and stress\)\.ModelMappingRedditΔ\\DeltaAUC%TwitterΔ\\DeltaAUC%BERTA — 4\-class \(primary\)−34\.5%\-34\.5\\%−39\.5%\-39\.5\\%BERTB — Binary \(normal vs\. MH\)−34\.3%\-34\.3\\%−37\.2%\-37\.2\\%BERTC — 3\-class \(normal/dep/distress\)−36\.3%\-36\.3\\%−39\.0%\-39\.0\\%BERTD — Distress superclass−34\.3%\-34\.3\\%−37\.2%\-37\.2\\%RoBERTaA — 4\-class \(primary\)−35\.4%\-35\.4\\%−38\.9%\-38\.9\\%RoBERTaB — Binary \(normal vs\. MH\)−34\.7%\-34\.7\\%−36\.1%\-36\.1\\%RoBERTaC — 3\-class−37\.0%\-37\.0\\%−38\.4%\-38\.4\\%RoBERTaD — Distress superclass−34\.7%\-34\.7\\%−36\.1%\-36\.1\\%Emotion\-DistilRoBERTaA — 4\-class−30\.3%\-30\.3\\%−37\.9%\-37\.9\\%Emotion\-DistilRoBERTaB — Binary−30\.1%\-30\.1\\%−38\.8%\-38\.8\\%Emotion\-DistilRoBERTaC — 3\-class−31\.8%\-31\.8\\%−38\.3%\-38\.3\\%Emotion\-DistilRoBERTaD — Distress−30\.1%\-30\.1\\%−38\.8%\-38\.8\\%GoEmotions\-RoBERTaA — 4\-class−28\.6%\-28\.6\\%−38\.6%\-38\.6\\%GoEmotions\-RoBERTaB — Binary−26\.8%\-26\.8\\%−35\.3%\-35\.3\\%GoEmotions\-RoBERTaC — 3\-class−29\.6%\-29\.6\\%−37\.6%\-37\.6\\%GoEmotions\-RoBERTaD — Distress−26\.8%\-26\.8\\%−35\.3%\-35\.3\\%![Refer to caption](https://arxiv.org/html/2608.26138v1/fig11_sensitivity_auc.jpg)Figure 11:Cross\-platformaucunder four label mapping schemes\. Each bar shows meanaucwith95%95\\%bootstrap confidence interval\.
### 5\.10Truncation and Text Length

Kaggle samples have a mean word count of118\.3118\.3\(median67\.067\.0;p75=154\.0p\_\{75\}\{=\}154\.0\) compared with13\.513\.5\(Reddit\) and19\.019\.0\(Twitter\)\. An estimated59\.2%59\.2\\%of Kaggle samples exceed the6464\-token maximum sequence length \(n=4,508n\{=\}4\{,\}508\), versus0\.0%0\.0\\%of Reddit and2\.8%2\.8\\%of Twitter samples\. Models trained on truncated long\-form Kaggle text have been exposed to clinical narrative structure — symptom trajectories, temporal references, and first\-person disclosures — that is structurally absent from short\-form Reddit comments and tweets\. This platform asymmetry supports the domain\-shift interpretation: the cross\-platform failure reflects genuine distributional divergence in communicative genre and text register — a form of covariate shift — rather than merely a consequence of label\-mapping heuristics\.

## 6Discussion

Across all fivecpfeaxes, the results converge on a structural interpretation: the observed cross\-platform failure patterns are not specific to any single evaluated transformer architecture, and are consistent with a general property of single\-platform proxy\-label training\. These failures co\-occur across all three independently evaluated models — degraded discrimination, miscalibration, attribution shift, and prediction rate disparities manifest simultaneously on both cross\-platform test sets — and this consistent co\-occurrence across architecturally diverse models is the study’s central empirical observation\.

### 6\.1The Scale and Consistency of Cross\-Platform Failure

Moving fromauc0\.9830\.983–0\.9870\.987toauc0\.5960\.596–0\.7030\.703represents a severe loss of discriminative utility\.aucvalues of0\.5960\.596–0\.7030\.703are above chance \(auc=0\.50\\text\{\{auc\}\{\}\}=0\.50\) and retain some signal, but fall below theauc≥0\.80\\text\{\{auc\}\{\}\}\\geq 0\.80threshold commonly cited as a minimum for discriminative utility in clinical classification tasks\[[31](https://arxiv.org/html/2608.26138#bib.bib31),[32](https://arxiv.org/html/2608.26138#bib.bib32)\]— though that benchmark was established with validated clinical outcomes, not proxy labels\. The F1\-macro collapse from0\.8620\.862–0\.8830\.883to0\.2840\.284–0\.3320\.332means that even a basic recall\-precision balance across classes is unachievable on cross\-platform data\.ecevalues of0\.4990\.499–0\.5420\.542on Twitter indicate that model\-expressed probabilities deviate substantially from true class probabilities \(by approximately5050percentage points on average\)\.

The consistency of these findings across three independently evaluated architecturally diverse models \(with GoEmotions\-RoBERTa providing a non\-independent partial reference\) suggests the failure is not specific to any particular model, but of the distributional mismatch between single\-platform proxy\-label training data and heterogeneous deployment environments\. As noted, this benchmark was developed for validated clinical endpoints; it is reported here only to convey the magnitude of degradation relative to a conventional reference, not as a directly applicable threshold for proxy\-label classification\.

Among the four evaluated models, GoEmotions\-RoBERTa shows the smallest Redditaucdrop \(28\.6%28\.6\\%\), but this reflects pretraining overlap with the specific test domain rather than acquired cross\-platform robustness; the advantage does not extend to Twitter \(drop38\.6%38\.6\\%\), where all four models perform similarly\.

### 6\.2Temperature Scaling as a Practical Post\-Hoc Calibration Remedy

Temperature scaling provides an immediate post\-hoc response to calibration failure that requires no model retraining but does require a small labelled calibration sample from the target platform \(n=625n\{=\}625for Reddit;n=286n\{=\}286for Twitter\)\. As the fine\-tuning experiment in Section[5\.8](https://arxiv.org/html/2608.26138#S5.SS8)shows, when labelled target\-platform samples are available, model retraining recovers discriminative performance substantially more effectively than post\-hoc recalibration\. Temperature scaling remains the appropriate remedy when retraining is computationally infeasible\.

### 6\.3Construct Validity: Emotion Labels and Clinical Categories

The cross\-platform test sets use emotion\-annotated data remapped to clinical categories\. The construct validity concern — that anger≠\\neqstress and fear≠\\neqanxiety clinically — is directly addressed by Mapping D \(distress superclass\)\. Under this mapping, which eliminates the anxiety/stress distinction and tests only normal vs\. depression vs\. distress, degradation is statistically indistinguishable from the primary analysis on both Reddit \(drops26\.826\.8–34\.3%34\.3\\%\) and Twitter \(drops35\.335\.3–39\.0%39\.0\\%\)\. The observed failure is therefore a property of cross\-platform distribution shift, not of the label mapping\.

### 6\.4Cross\-Platform Deployment Equity Implications

A model with depression proxy\-classdi=0\.133\\textsc\{di\}\{=\}0\.133\(RoBERTa on Reddit\) generates depression proxy\-class predictions at approximately one\-seventh the rate observed on the Kaggle reference platform\. The prior\-shift\-adjusteddiof0\.110\.11–0\.290\.29confirms that a substantial disparity persists beyond what prevalence differences alone explain\. Theeodvalues of0\.7530\.753–0\.8300\.830for depression, anxiety, and stress on Reddit indicate that the conditional true positive rate drops by7575–8383percentage points relative to within\-platform performance\.

These findings motivate treating platform\-stratified prediction equity metrics as an informative diagnostic dimension during pre\-deployment evaluation, not as definitive evidence of algorithmic bias\. The appropriate response to largediandeodvalues is platform\-specific model validation and potential recalibration\.

### 6\.5Illustrative CPFE Degradation Benchmarks

In this study, all three independently evaluated models showedaucdrops exceeding30%30\\%on both platforms, a magnitude associated with near\-chance minority\-class performance\. These figures derive from one training corpus, three platforms, four models, and proxy emotion labels; they should be treated as descriptive baselines for future cross\-platform evaluation work rather than as validated decision thresholds\.

## 7Limitations

1. 1\.Construct validity of cross\-platform labels\.The emotion→\\toclinical label mappings are heuristic approximations\. While Mapping D provides evidence of robustness to the most salient construct validity concern, human\-annotated clinical labels on Reddit and Twitter would be required to fully disentangle construct mismatch from distributional failure\.
2. 2\.Pretraining contamination\.GoEmotions\-RoBERTa and Emotion\-DistilRoBERTa have pretraining overlap with the evaluation domains\. Their Reddit performance figures are partially non\-independent and should be treated as upper bounds\.
3. 3\.Attribution method reliability\.Token importance was estimated by gradient\-based saliency\[[25](https://arxiv.org/html/2608.26138#bib.bib25)\], a local linear approximation to feature importance\. This method has known limitations: it can be sensitive to input normalisation and random seed, and its faithfulness to model decision logic is not guaranteed\[[27](https://arxiv.org/html/2608.26138#bib.bib27),[28](https://arxiv.org/html/2608.26138#bib.bib28)\]\. Multiple studies have demonstrated that gradient saliency rankings can be inconsistent across seeds and are not always faithful to model decision logic\. The near\-zero Jaccard values should be interpreted as consistent with substantial attribution instability rather than as definitive evidence of attribution collapse\. Confirmation with Shapley\-based or Integrated Gradients methods\[[34](https://arxiv.org/html/2608.26138#bib.bib34)\]is a priority for future work\.
4. 4\.Class imbalance and prior shift\.The Kaggle partition is heavily skewed toward depression \(56\.6%56\.6\\%\) relative to Reddit \(6\.6%6\.6\\%\) and Twitter \(30\.1%30\.1\\%\)\. Separate estimation of prior shift vs\. likelihood shift contributions would strengthen causal attribution of the observed cross\-platform prediction equity disparities\.
5. 5\.Residual variance in secondary metrics across seeds\.Whileaucdegradation was consistent across all five seeds, secondary metrics — specifically Symmetric Disparate Impact, Equalized Odds Difference, and Jaccard similarity scores — were computed only for the primary seed \(seed=\{=\}42\); seed\-level variance for these metrics was not characterised\. Multi\-seed replication of equity and attribution metrics is a priority for future work\.
6. 6\.Fine\-tuning comparison\.The meanaucgain of0\.2160\.216is a point estimate without characterised variance \(single seed; seed=42=42\)\. Multi\-seed replication of the fine\-tuning experiment is a priority for future work\.
7. 7\.Temperature scaling calibration split from test data\.The optimal temperatureT∗T^\{\*\}was estimated on a10%10\\%stratified split from the target platform test set; post\-calibrationecevalues are therefore potentially optimistic relative to a fully novel deployment sample\.
8. 8\.Absence of non\-transformer and zero\-shot baselines\.It is unknown whether classical models \(logistic regression, SVM\) or large zero\-shot language models \(GPT\-series\) exhibit the same cross\-platform degradation patterns\. These comparisons would help establish whether the observed failure is specific to fine\-tuned transformers or a more general property of cross\-platform text classification\.

## 8Conclusion

Thecpfeframework — a five\-axis audit covering discriminative performance, calibration, statistical significance, prediction equity, and attribution stability — was applied to four transformer\-based classifiers trained on a Kaggle mental health corpus and evaluated cross\-platform on Reddit and Twitter using remapped proxy labels \(see Section[3](https://arxiv.org/html/2608.26138#S3)\)\. Across three independently evaluated models \(BERT, RoBERTa, Emotion\-DistilRoBERTa\) and five training seeds, all five axes revealed co\-occurring failures:aucfell30\.330\.3–35\.4%35\.4\\%on Reddit and37\.937\.9–39\.5%39\.5\\%on Twitter;ecerose from0\.0560\.056–0\.0600\.060within\-platform to0\.4990\.499–0\.5420\.542on Twitter; gradient attribution token overlap was near\-zero \(J=0J\{=\}0in14/1614/16model–class pairs atK=10K\{=\}10\); and cross\-platform prediction rate disparities severely violated the four\-fifths rule\[[16](https://arxiv.org/html/2608.26138#bib.bib16)\]\.

Temperature scaling substantially improved calibration \(meanecereduction88\.0%88\.0\\%\) without restoring discrimination; a single\-seed target\-domain fine\-tuning experiment on the same10%10\\%labelled split produced a meanaucgain of0\.2160\.216\(excluding an outlier pair,0\.1970\.197\), suggesting that available target\-platform labels may provide greater benefit as training signal than as calibration signal\. These results were consistent across four label mapping schemes and five independent training seeds, providing strong empirical support for the view that cross\-platform validation across all fivecpfeaxes should be treated as a standard requirement for mental healthnlpsystems intended for heterogeneous deployment environments\.

## Author Contributions

Conceptualisation: R\.S\.P\., S\.Y\. Methodology: R\.S\.P\., S\.Y\. Software: R\.S\.P\. Formal analysis: R\.S\.P\., S\.Y\. Investigation: R\.S\.P\., S\.Y\. Data curation: R\.S\.P\. Writing — original draft: R\.S\.P\. Writing — review & editing: R\.S\.P\., S\.Y\. Visualisation: R\.S\.P\.

## Data and Code Availability

## Declaration of Competing Interests

The authors declare no competing financial or non\-financial interests\.

## References

- \[1\]S\. Chancellor and M\. De Choudhury, “Methods in predictive techniques for mental health status on social media: A critical review,”npj Digital Medicine, vol\. 3, no\. 1, p\. 43, 2020\.
- \[2\]G\. Coppersmith, M\. Dredze, and C\. Harman, “Quantifying mental health signals in Twitter,” inProc\. ACL Workshop on Computational Linguistics and Clinical Psychology, 2014, pp\. 51–60\.
- \[3\]J\. Eisenstein, “What to do about bad language on the internet,” inProc\. NAACL\-HLT, 2013, pp\. 359–369\.
- \[4\]M\. De Choudhury, M\. Gamon, S\. Counts, and E\. Horvitz, “Predicting depression via social media,” inProc\. Int\. AAAI Conf\. Web and Social Media, vol\. 7, 2013, pp\. 128–137\.
- \[5\]S\. Ji, S\. Pan, X\. Li, E\. Cambria, G\. Long, and Z\. Huang, “Suicidal ideation detection: A review of machine learning methods and applications,”IEEE Trans\. Comput\. Social Syst\., vol\. 9, no\. 1, pp\. 243–258, 2022\.
- \[6\]A\. Cohan, B\. Desmet, A\. Yates, L\. Soldaini, S\. MacAvaney, and N\. Goharian, “SMHD: A large\-scale resource for exploring online language usage for multiple mental health conditions,” inProc\. COLING, 2018, pp\. 1485–1497\.
- \[7\]E\. W\. Steyerberg, K\. G\. M\. Moons, D\. A\. van der Windt,et al\., “Prognosis Research Strategy \(PROGRESS\) 3: Prognostic model research,”PLOS Medicine, vol\. 10, no\. 2, p\. e1001381, 2013\.
- \[8\]G\. S\. Collins, J\. B\. Reitsma, D\. G\. Altman, and K\. G\. M\. Moons, “Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis \(TRIPOD\),”Ann\. Internal Medicine, vol\. 162, no\. 1, pp\. 55–63, 2015\.
- \[9\]D\. Demszky, D\. Movshovitz\-Attias, J\. Ko, A\. Cowen, G\. Nemade, and S\. Ravi, “GoEmotions: A dataset of fine\-grained emotions,” inProc\. 58th Annual Meeting ACL, 2020, pp\. 4040–4054\.
- \[10\]E\. Saravia, H\.\-C\. T\. Liu, Y\.\-H\. Huang, J\. Wu, and Y\.\-S\. Chen, “CARER: Contextualized affect representations for emotion recognition,” inProc\. EMNLP, 2018, pp\. 3687–3697\.
- \[11\]J\. Blitzer, M\. Dredze, and F\. Pereira, “Biographies, Bollywood, boom\-boxes and blenders: Domain adaptation for sentiment classification,” inProc\. ACL, 2007, pp\. 440–447\.
- \[12\]M\. Kula, L\. Baldini Soares, and J\. Eisenstein, “Towards cross\-platform generalisation of mental health content classifiers,”arXiv:2205\.05521, 2022\.
- \[13\]A\. Romanov and C\. Shivade, “Lessons from natural language inference in the clinical domain,” inProc\. EMNLP, 2018, pp\. 1586–1596\.
- \[14\]S\. Gururangan, A\. Marasović, S\. Swayamdipta,et al\., “Don’t stop pretraining: Adapt language models to domains and tasks,” inProc\. ACL, 2020, pp\. 8342–8360\.
- \[15\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger, “On calibration of modern neural networks,” inProc\. ICML, PMLR 70, 2017, pp\. 1321–1330\.
- \[16\]S\. Barocas, M\. Hardt, and A\. Narayanan,Fairness and Machine Learning: Limitations and Opportunities\. fairmlbook\.org, 2019\.
- \[17\]M\. Hardt, E\. Price, and N\. Srebro, “Equality of opportunity in supervised learning,” inAdvances in NeurIPS, vol\. 29, 2016, pp\. 3315–3323\.
- \[18\]S\. L\. Blodgett, S\. Barocas, H\. Daumé III, and H\. Wallach, “Language \(technology\) is power: A critical survey of ‘bias’ in NLP,” inProc\. ACL, 2020, pp\. 5454–5476\.
- \[19\]S\. Sarkar, “Combined Mental Health Dataset,” Kaggle, 2022\. \[Online\]\. Available:[https://www\.kaggle\.com/datasets/suchintikasarkar/sentiment\-analysis\-for\-mental\-health](https://www.kaggle.com/datasets/suchintikasarkar/sentiment-analysis-for-mental-health)
- \[20\]I\. Loshchilov and F\. Hutter, “Decoupled weight decay regularization,” inProc\. ICLR, 2019\.
- \[21\]J\. Devlin, M\.\-W\. Chang, K\. Lee, and K\. Toutanova, “BERT: Pre\-training of deep bidirectional transformers for language understanding,” inProc\. NAACL\-HLT, 2019, pp\. 4171–4186\.
- \[22\]Y\. Liu, M\. Ott, N\. Goyal,et al\., “RoBERTa: A robustly optimized BERT pretraining approach,”arXiv:1907\.11692, 2019\.
- \[23\]J\. Hartmann, “Emotion English DistilRoBERTa\-base,” Hugging Face, 2022\. \[Online\]\. Available:[https://huggingface\.co/j\-hartmann/emotion\-english\-distilroberta\-base](https://huggingface.co/j-hartmann/emotion-english-distilroberta-base)
- \[24\]E\. R\. DeLong, D\. M\. DeLong, and D\. L\. Clarke\-Pearson, “Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach,”Biometrics, vol\. 44, no\. 3, pp\. 837–845, 1988\.
- \[25\]K\. Simonyan, A\. Vedaldi, and A\. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” inProc\. ICLR Workshop, 2014\.
- \[26\]J\. Quiñonero\-Candela, M\. Sugiyama, A\. Schwaighofer, and N\. D\. Lawrence, \(Eds\.\),Dataset Shift in Machine Learning\. Cambridge, MA: MIT Press, 2009\.
- \[27\]J\. Adebayo, J\. Gilmer, M\. Muelly, I\. Goodfellow, M\. Hardt, and B\. Kim, “Sanity checks for saliency maps,” inAdvances in NeurIPS, vol\. 31, 2018, pp\. 9505–9515\.
- \[28\]S\. Jain and B\. C\. Wallace, “Attention is not explanation,” inProc\. NAACL\-HLT, 2019, pp\. 3543–3556\.
- \[29\]Z\. Obermeyer, B\. Powers, C\. Vogeli, and S\. Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations,”Science, vol\. 366, no\. 6464, pp\. 447–453, 2019\.
- \[30\]I\. Y\. Chen, P\. Szolovits, and M\. Ghassemi, “Can AI help reduce disparities in general medical and mental health care?”AMA Journal of Ethics, vol\. 22, no\. 2, pp\. E167–E179, 2020\.
- \[31\]C\. E\. Metz, “Basic principles of ROC analysis,”Seminars in Nuclear Medicine, vol\. 8, no\. 4, pp\. 283–298, 1978\.
- \[32\]M\. H\. Zweig and G\. Campbell, “Receiver\-operating characteristic \(ROC\) plots: A fundamental evaluation tool in clinical medicine,”Clinical Chemistry, vol\. 39, no\. 4, pp\. 561–577, 1993\.
- \[33\]N\. Kokhlikyan, V\. Miglani, M\. Martin,et al\., “Captum: A unified and generic model interpretability library for PyTorch,”arXiv:2009\.07896, 2020\.
- \[34\]M\. Sundararajan, A\. Taly, and Q\. Yan, “Axiomatic attribution for deep networks,” inProc\. 34th ICML, PMLR 70, 2017, pp\. 3319–3328\.

## Appendix ASupplementary Figures

The following supplementary figures are provided to support the main text\.

![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s1_perclass_auc.jpg)Figure 12:Supplementary Fig\. S1: Per\-class one\-vs\-restaucwith95%95\\%DeLong confidence intervals across all four models and three platforms \(4848estimates\)\. The red dashed line marks chance performance \(auc=0\.50=0\.50\)\. Stress class CIs on Twitter approach or touch the chance boundary for BERT \(0\.5300\.530\[0\.5000\.500,0\.5600\.560\]\) and RoBERTa \(0\.5220\.522\[0\.4920\.492,0\.5510\.551\]\)\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s2_ece_bootstrap.jpg)Figure 13:Supplementary Fig\. S2: Expected Calibration Error \(ece\) with95%95\\%bootstrap CIs \(B=1,000B\{=\}1\{,\}000\) across all models and platforms\. Within\-platform Kaggleeceis consistently below0\.060\.06; Redditeceis0\.1960\.196–0\.2290\.229; Twittereceexceeds0\.4990\.499–0\.5420\.542for all models\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s3_perclass_ece.jpg)Figure 14:Supplementary Fig\. S3: Aggregateece\(left panel\) and per\-class one\-vs\-restecewith bootstrap95%95\\%CIs \(right panel\),B=1,000B\{=\}1\{,\}000\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s4_eod_heatmap.jpg)Figure 15:Supplementary Fig\. S4: Equalized Odds Difference \(eod\) heatmap across all four models, platforms, and classes\.eodfor Depression on Reddit reaches0\.8070\.807–0\.8300\.830across all models\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s5_vocab_retention.jpg)Figure 16:Supplementary Fig\. S5: Proxy\-clinical vocabulary retention in top\-10 attribution tokens\. Each cell shows the fraction of the top\-10 gradient saliency tokens that match a predefined expert proxy\-clinical vocabulary list\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s6_jaccard_avg.jpg)Figure 17:Supplementary Fig\. S6: Average Jaccard similarity of top\-10 attribution words across both cross\-platform comparisons, by model and class\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s7_jaccard_k_sensitivity.jpg)Figure 18:Supplementary Fig\. S7: Jaccard feature stability sensitivity toKK\(number of top\-attributed tokens\), for Kaggle→\\toReddit comparison\. Shaded bands show range across four models\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s8_auc_heatmap_mappings.jpg)Figure 19:Supplementary Fig\. S8: Macroaucheatmap across all models, platforms, and label mapping schemes \(A–D\)\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s9_text_length.jpg)Figure 20:Supplementary Fig\. S9: Text length distributions across platforms \(violin plots\)\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s10_auc_length_quartile.jpg)Figure 21:Supplementary Fig\. S10: Macroaucstratified by text\-length quartile \(Q1: shortest25%25\\%; Q4: longest25%25\\%\)\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s11_attr_roberta_stress.jpg)Figure 22:Supplementary Fig\. S11: Cross\-platform gradient token attribution for RoBERTa, Stress class\. Kaggle features include partially proxy\-clinical tokens \(weighted, stressed, care\); Reddit and Twitter features shift to platform\-specific conversational vocabulary\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s12_attr_edistilroberta_stress.jpg)Figure 23:Supplementary Fig\. S12: Cross\-platform gradient token attribution for Emotion\-DistilRoBERTa, Stress class\.![Refer to caption](https://arxiv.org/html/2608.26138v1/fig_s13_attr_goemotions_stress.jpg)Figure 24:Supplementary Fig\. S13: Cross\-platform gradient token attribution for GoEmotions\-RoBERTa, Stress class\. All three platform vocabularies are fully disjoint \(J=0\.000J\{=\}0\.000for this model\-class pair on Kaggle\-to\-Twitter comparison\)\.