Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4134–4145 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4134 Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting Guanhua Zhang‡§∗, Bing Bai§∗, Junqi Zhang§, Kun Bai§, Conghui Zhu‡, Tiejun Zhao‡ ‡Harbin Institute of Technology, §Tencent CSIG {guanhzhang,icebai,benjqzhang,kunbai}@tencent.com, {chzhu,tjzhao}@hit-mtlab.net Abstract With the recent proliferation of the use of text classifications, researchers have found that there are certain unintended biases in text clas- sification datasets. For example, texts con- taining some demographic identity-terms (e.g., “gay”, “black”) are more likely to be abusive in existing abusive language detection datasets. As a result, models trained with these datasets may consider sentences like “She makes me happy to be gay” as abusive simply because of the word “gay.” In this paper, we formal- ize the unintended biases in text classification datasets as a kind of selection bias from the non-discrimination distribution to the discrim- ination distribution. Based on this formal- ization, we further propose a model-agnostic debiasing training framework by recovering the non-discrimination distribution using in- stance weighting, which does not require any extra resources or annotations apart from a pre-defined set of demographic identity-terms. Experiments demonstrate that our method can effectively alleviate the impacts of the un- intended biases without significantly hurting models’ generalization ability. 1 Introduction With the development of Natural Language Pro- cessing (NLP) techniques, Machine Learning (ML) models are being applied in continuously expand- ing areas (e.g., to detect spam emails, to filter resumes, to detect abusive comments), and they are affecting everybody’s life from many aspects. However, human-generated datasets may intro- duce some human social prejudices to the mod- els (Caliskan-Islam et al., 2016). Recent works have found that ML models can capture, utilize, and even amplify the unintended biases (Zhao et al., 2017), which has raised lots of concerns about the ∗ Equal contributions from both authors. This work was done when Guanhua Zhang was an intern at Tencent. Identity-term Count Percentage Toxic gay 868 57.4% homosexual 202 34.4% Mexican 116 21.6% blind 257 14.8% black 1,123 13.1% overall 159,686 9.6% Table 1: Percentage of toxic comments by some spe- cific demographic identity-terms in the dataset released by Dixon et al. (2018). discrimination problem in NLP models (Sun et al., 2019). Text classification is one of the fundamental tasks in NLP. It aims at assigning any given sen- tence to a specific class. In this task, models are expected to make predictions with the semantic in- formation rather than with the demographic group identity information (e.g., “gay”, “black”) con- tained in the sentences. However, recent research points out that there widely exist some unintended biases in text clas- sification datasets. For example, in a toxic com- ment identification dataset released by Dixon et al. (2018), it is found that texts containing some spe- cific identity-terms are more likely to be toxic. More specifically, 57.4% of comments containing “gay” are toxic, while only 9.6% of all samples …

Content truncated to 500 words