Perturbation Sensitivity Analysis to Detect Unintended Model Biases

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5740–5745, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5740 Perturbation Sensitivity Analysis to Detect Unintended Model Biases Vinodkumar Prabhakaran Google Brain San Francisco, CA, USA vinodkpg@google.com Ben Hutchinson Google Brain San Francisco, CA, USA benhutch@google.com Margaret Mitchell Google Brain Seattle, WA, USA mmitchellai@google.com Abstract Data-driven statistical Natural Language Pro- cessing (NLP) techniques leverage large amounts of language data to build models that can understand language. However, most lan- guage data reflect the public discourse at the time the data was produced, and hence NLP models are susceptible to learning inciden- tal associations around named referents at a particular point in time, in addition to gen- eral linguistic meaning. An NLP system de- signed to model notions such as sentiment and toxicity should ideally produce scores that are independent of the identity of such enti- ties mentioned in text and their social associa- tions. For example, in a general purpose senti- ment analysis system, a phrase such as I hate Katy Perry should be interpreted as having the same sentiment as I hate Taylor Swift. Based on this idea, we propose a generic evaluation framework, Perturbation Sensitivity Analysis, which detects unintended model biases related to named entities, and requires no new annota- tions or corpora. We demonstrate the utility of this analysis by employing it on two different NLP models — a sentiment model and a tox- icity model — applied on online comments in English language from four different genres. 1 Introduction Recent research has shown ample evidence that data-driven NLP models may inadvertently cap- ture, reflect and sometimes amplify various so- cial biases present in the language data they are trained on (Bolukbasi et al., 2016; Blodgett and O’Connor, 2017). Such biases can often result in unintended and disparate harms to the users who engage with NLP-aided systems. For instance, when NLP algorithms are used to moderate online communication, e.g. by detecting harassment, al- though the net social benefits may be positive, the harms caused by incorrect classifications may be unevenly distributed, leading to disparate impact (Feldman et al., 2015). Some writers may find Sentence Toxicity Sentiment I hate Justin Timberlake. 0.90 -0.30 I hate Katy Perry. 0.80 -0.10 I hate Taylor Swift. 0.74 -0.40 I hate Rihanna. 0.69 -0.60 Table 1: Sensitivity of NLP models to named entities in text. Toxicity score range: 0 to 1; Sentiment score range: -1 to +1. their contributions being disproportionately cen- sored, while some readers may not be adequately protected from harassment (Dixon et al., 2018). Research into fairness in machine learning dis- tinguishes two broad categories of unfair discrim- ination. First, unfairness for individuals exists when similar individuals are treated dissimilarly (Dwork et al., 2012). Second, a range of criteria define unfairness for groups, each in terms of sta- tistical dependence between group membership, model score, and class label (see, e.g., (Choulde- chova and …

Content truncated to 500 words