MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7184–7190 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7184 MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare Shaoxiong Ji†, Tianlin Zhang‡, Luna Ansari†, Jie Fu§, Prayag Tiwari†, Erik Cambria¶ † Aalto University, Finland ‡ The University of Manchester, UK § Mila, Qu´ebec AI Institute, Canada ¶ Nanyang Technological University, Singapore {shaoxiong.ji; luna.ansari; prayag.tiwari}@aalto.fi tianlin.zhang@postgrad.manchester.ac.uk fujie@mila.quebec cambria@ntu.edu.sg Abstract Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and releases two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks. Keywords: Pretrained Language Models, Mental Healthcare, MentalBERT 1. Introduction Mental health is a global issue, especially severe in most developed countries and many emerging markets. According to the mental health action plan (2013 - 2020) from the World Health Organization, one in four people worldwide suffer from mental disorders to some extent. Moreover, three out of four people with severe mental disorders do not receive treatment, worsening the problem. During some periods like the pandemic, people struggle with mental health issues, and many may not get mental health practitioners’ help. Previous studies reveal that suicide risk usually has a connec- tion to mental disorders (Windfuhr and Kapur, 2011). Partly due to severe mental disorders, 900,000 people commit suicide each year worldwide, making suicide the second most common cause of death among the young. Suicide attempters have been reported as suf- fering from mental disorders, with an investigation on a shift from mental health to suicidal ideation conducted by language and interactional measures (De Choud- hury et al., 2016). Early identification is a practical approach to mental illness and suicidal ideation prevention. Except for traditional proactive screening, social media is a good channel for mental health care. Social media platforms such as Reddit and Twitter provide anonymous space for users to discuss stigmatic topics and self-report per- sonal issues. Social content from users who wrote about mental health issues and posted suicidal ideation has been widely used to study mental health issues (Ji et al., 2018; Tadesse et al., 2019). Machine learning- based detection techniques can empower healthcare workers in early detection and assessment to take an ac- tion of proactive prevention. Recent advances in deep learning facilitate the development of effective early detection methods (Cambria et al., 2010; Cambria …

Content truncated to 500 words