Gender and Racial Fairness in Depression Research using Social Media

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2932–2949 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2932 Gender and Racial Fairness in Depression Research using Social Media Carlos Aguirre, Keith Harrigian, Mark Dredze Johns Hopkins University caguirr4@jhu.edu, kharrigian@jhu.edu, mdredze@cs.jhu.edu Abstract Multiple studies have demonstrated that behav- ior on internet-based social media platforms can be indicative of an individual’s mental health status. The widespread availability of such data has spurred interest in mental health research from a computational lens. While pre- vious research has raised concerns about possi- ble biases in models produced from this data, no study has quantified how these biases actu- ally manifest themselves with respect to differ- ent demographic groups, such as gender and racial/ethnic groups. Here, we analyze the fair- ness of depression classifiers trained on Twit- ter data with respect to gender and racial de- mographic groups. We find that model per- formance systematically differs for underrep- resented groups and that these discrepancies cannot be fully explained by trivial data rep- resentation issues. Our study concludes with recommendations on how to avoid these biases in future research. 1 Introduction Work from De Choudhury et al. (2013) and Cop- persmith et al. (2014), showing that an individ- ual’s mental health can be evaluated based on the language they generate on social media platforms, has served as the basis for a substantial amount of computational research over the last decade. Subsequent studies have examined an even wider range of mental health conditions, social media platforms, and types of online behavior at both the individual and population level (Coppersmith et al., 2015b; Lynn et al., 2018; De Choudhury et al., 2016). Vast potential for societal benefits underlies this work, as conservative estimates suggest that 8.1% of American adults suffer from major depres- sive disorder at any given time and up to 16.2% of individuals will experience at least one major depressive episode during their lifetime (Kessler et al., 2003; Brody et al., 2018; Hasin et al., 2018). Mental health services are transitioning to online mediums at a rapid pace, with the recent COVID- 19 pandemic dramatically further accelerating this trend (Zhou et al., 2020; Ohannessian et al., 2020). Thus, analysis of online language may play a key role in mental health treatment in the future. Nonetheless, care must be taken to understand potential biases inherent in this research before any technologies are deployed in a clinical setting. For instance, previous work has found that Black and Hispanic/Latinx individuals are less likely to be treated for depression than White individuals (Simpson et al., 2007). Possibly a result of this underlying bias, recent studies of the US popu- lation have concluded that baseline rates of de- pression vary depending on demographics (Brody et al., 2018; Hasin et al., 2018) — major depres- sive disorder was found to be more prevalent in females and White adults. Yet, it remains unclear whether these supposed differences in depression prevalence between gender and …

Content truncated to 500 words