Do Models of Mental Health Based on Social Media Data Generalize?

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3774–3788 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3774 Do Models of Mental Health Based on Social Media Data Generalize? Keith Harrigian, Carlos Aguirre, Mark Dredze Johns Hopkins University kharrigian@jhu.edu, caguirr4@jhu.edu, mdredze@cs.jhu.edu Abstract Proxy-based methods for annotating mental health status in social media have grown popu- lar in computational research due to their abil- ity to gather large training samples. However, an emerging body of literature has raised new concerns regarding the validity of these types of methods for use in clinical applications. To further understand the robustness of distantly supervised mental health models, we explore the generalization ability of machine learning classifiers trained to detect depression in indi- viduals across multiple social media platforms. Our experiments not only reveal that substan- tial loss occurs when transferring between plat- forms, but also that there exist several unreli- able confounding factors that may enable re- searchers to overestimate classification perfor- mance. Based on these results, we enumer- ate recommendations for future mental health dataset construction. 1 Introduction In the last decade, there has been substantial growth in the area of digital psychiatry. Automated meth- ods using natural language processing have been able to detect mental health disorders based on a person’s language in a variety of data types, such as social media (Mowery et al., 2016; Morales et al., 2017), speech (Iter et al., 2018) and other writings (Kayi et al., 2017; Just et al., 2019). As in-person clinical visits are made increasingly difficult by so- cioeconomic barriers and public-health crises, such as COVID-19, tools for measuring mental wellness using implicit signal become more important than ever (Abdel-Rahman, 2019; Bojdani et al., 2020). Early work in this area leveraged traditional hu- man subject studies in which individuals with clini- cally validated psychiatric diagnoses volunteered their language data to train classifiers and perform quantitative analyses (Rude et al., 2004; Jarrold et al., 2010). In an effort to model larger, more di- verse populations with less overhead, a substantial portion of research in the last decade has instead explored data annotated via automated mechanisms (Coppersmith et al., 2015a; Winata et al., 2018). Studies leveraging proxy-based annotations have supported their design by demonstrating alignment with existing psychological theory regarding lan- guage usage by individuals living with a mental health disorder (Cavazos-Rehg et al., 2016; Vedula and Parthasarathy, 2017). For example, feature analyses have highlighted higher amounts of neg- ative affect and increased personal pronoun preva- lence amongst depressed individuals (Park et al., 2012; De Choudhury et al., 2013). Given these con- sistencies, the field has largely turned its attention toward optimizing predictive power via state of the art models (Orabi et al., 2018; Song et al., 2018). The ultimate goal of these efforts has been threefold—to better personalize psychiatric care, to enable early intervention, and to monitor population-level health outcomes in real time. Nonetheless, research has largely trudged forward without stopping to ask one critical question: do models of mental health …

Content truncated to 500 words