Detecting and Reducing Bias in a High Stakes Domain

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4765–4775, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4765 Detecting and Reducing Bias in a High Stakes Domain Ruiqi Zhong1 ⇤, Yanda Chen1, Desmond Patton2, Charlotte Selous2, Kathleen McKeown1 1Department of Computer Science, Columbia University 2School of Social Work, Columbia University {rz2383, yc3384, dp2787, cis2114}@columbia.edu {kathy}@cs.columbia.edu Abstract Gang-involved youth in cities such as Chicago sometimes post on social media to express their aggression towards rival gangs and pre- vious research has demonstrated that a deep learning approach can predict aggression and loss in posts. To address the possibility of bias in this sensitive application, we developed an approach to systematically interpret the state of the art model. We found, surprisingly, that it frequently bases its predictions on stop words such as “a” or “on”, an approach that could harm social media users who have no aggres- sive intentions. To tackle this bias, domain experts annotated the rationales, highlighting words that explain why a tweet is labeled as “aggression”. These new annotations enable us to quantitatively measure how justified the model predictions are, and build models that drastically reduce bias. Our study shows that in high stake scenarios, accuracy alone can- not guarantee a good system and we need new evaluation methods. 1 Introduction There has been increased interest in using natu- ral language processing to help understand the root causes of violence in cities such as Chicago. Re- searchers in social work found that posts on so- cial media by gang-involved youth mirror their life on the physical street (Patton et al., 2016, 2017) and suggested that if community outreach workers could use tools to help identify posts of aggres- sion and grief, they could intervene before grief turns to retribution and help avert violence. Early research in this area (Blevins et al., 2016) used support vector machines for the three-way clas- sification task of aggression, loss and other and recent work (Chang et al., 2018) demonstrated a ⇤The first author is now a PhD student at University of California, Berkeley. significant increase in accuracy through the use of a neural net approach along with the representa- tion of contextual information from previous con- versations. In both of these approaches, computer scientists and social work researchers worked side by side to enable qualitative analysis of the data to use in machine learning and to provide meaningful error analyses. Given the interest in using this work in a live setting to identify situations where commu- nity outreach workers could intervene, we are concerned with how the model makes the deci- sion. This is especially important since this is a high stakes domain and machine learning sys- tems are known to incorporate bias. For exam- ple, the COMPAS Recidivism Algorithm, which assesses the likelihood of a criminal defendant to re-offend and influences judges’ sentencing deci- sions, is known to be biased against African Amer- icans …

Content truncated to 500 words