Fine-Tuned RoBERTa Transformers for Multi-Category Mental Health Status Recognition in Written Text

Abstract

Detecting early signs of anxiety, depression, and stress in regular texts may prove tremendously helpful for the triage process and for reducing the clinician's case-managing load. This paper describes the creation and tuning of a transformer-based classifier on short self-reported statements sorted into seven mental-health categories: Anxiety, Bipolar, Depression, Normal, Personality Disorder, Stress, and Suicidal. We use the pretrained RoBERTa-base weights as an initializer for a transformer-based classifier with a linear classification head on top of the base model. We trained the classifier on a balanced set of 105000 annotated statements, 15000 for each category, and evaluated its performance on a separate held-out sample of 9767 previously unseen statements. It demonstrated outstanding results, achieving 93.87% accuracy on the test set with a mean F1-score of 0.9408 and a per-category F1-score varying from 0.808 (for Depression) to 0.997 (for Personality Disorder). Upon a closer inspection of misclassified texts, we noticed an intriguing pattern: for 80.6% of misclassified cases, the true category was in the vicinity of Depression. Our investigation suggested that this might be explained by either the confounding presence of similar features in both categories or by the classifier's bias. For a closely related task, a classical ML pipeline with gradient-boosted trees was reported by other authors to achieve around 84.2% accuracy on a coarser, three-category version of this problem. Using the same underlying data but more than doubling the classification resolution to seven categories, our fine-tuned transformer achieved a much better overall accuracy. Finally, a proof of concept web-based application was created to showcase the potential of such technology. Overall, our experiments demonstrated that there is a realistic possibility of using a fine-tuned transformer for mental health screening.