TBD3: A Thresholding-Based Dynamic Depression Detection from Social Media for Low-Resource Users

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2157–2165 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2157 TBD3: Thresholding-Based Dynamic Depression Detection from Social Media for Low-Resource Users Hrishikesh Kulkarni1, Sean MacAvaney2, Nazli Goharian3, Ophir Frieder3 1 Georgetown University, 2 University of Glasgow, 3 IR Lab, Georgetown University 1 Washington, DC, USA, 2 Glasgow, UK, 3 Washington, DC, USA 1 hpk8@georgetown.edu, 2 first.last@glasgow.ac.uk, 3 first@ir.cs.georgetown.edu Abstract Social media are heavily used by many users to share their mental health concerns and diagnoses. This trend has turned social media into a large-scale resource for researchers focused on detecting mental health conditions. Social media usage varies considerably across individuals. Thus, classification of patterns, including detecting signs of depression, must account for such variation. We address the disparity in classification effectiveness for users with little activity (e.g., new users). Our evaluation, performed on a large-scale dataset, shows considerable detection discrepancy based on user posting frequency. For instance, the F1 detection score of users with an above-median versus below-median number of posts is greater than double (0.803 vs 0.365) using a conventional CNN-based model; similar results were observed on lexical and transformer-based classifiers. To complement this evaluation, we propose a dynamic thresholding technique that adjusts the classifier’s sensitivity as a function of the number of posts a user has. This technique alone reduces the margin between users with many and few posts, on average, by 45% across all methods and increases overall performance, on average, by 33%. These findings emphasize the importance of evaluating and tuning natural language systems for potentially vulnerable populations. Keywords: depression detection; social media; thresholding; mental health; early risk detection 1. Introduction Depression affects between 3% (UN, 2017) and 5% (WHO, 2021) of the global population and can lead to a variety of negative outcomes including suicide, self-harm, dementia, and premature mortality. Hence, early depression detection is crucial for preventing ex- acerbation of the condition. Best performing mental health monitoring methods are highly data-driven, of- ten bound by the pertinence and volume of the dataset (Harrigian et al., 2021). Reaching actionable accuracy is particularly difficult when a user has a short posting history, providing systems little data to make an assess- ment. We investigate the identification of depression in users with short posting histories (which we call low- resource users). We provide an evaluation of the per- formance of low-resource users (relative to their high- resource counterparts) using an existing large-scale dataset: the Reddit Self-reported Depression Diagno- sis dataset (Yates et al., 2017, RSDD). By stratifying the dataset by the number of posts a user has, we find that a variety of classifiers perform substantially worse on low-resource users. For instance, the F1 score of a CNN-based model is 0.803 for high-resource users, but only 0.365 for their low-resource counterparts. Recognizing that confounding factors could affect the aforementioned evaluation, we produce a new resource that aims to better control for the textual content across lower- and higher-resource users. This …

Content truncated to 500 words