pre-trained-language-models

This paper presents MentalMARBERT, a domain-adapted Arabic language model for detecting mental health disorders from social media text. The framework uses domain-adaptive pre-training and a two-stage fine-tuning approach, achieving 0.877 accuracy and 0.861 macro-F1 on a newly constructed Arabic mental health dataset of 50,670 tweets.

This paper proposes an attention expansion mechanism to enhance keyphrase extraction from long documents by augmenting PLM token representations with out-of-context information, achieving consistent improvements over state-of-the-art models without requiring full-document attention or expensive LLM inference.