Fortifying Toxic Speech Detectors Against Veiled Toxicity

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7732–7739, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7732 Fortifying Toxic Speech Detectors Against Veiled Toxicity Xiaochuang Han Carnegie Mellon University xiaochuh@cs.cmu.edu Yulia Tsvetkov Carnegie Mellon University ytsvetko@cs.cmu.edu Abstract Modern toxic speech detectors are incompe- tent in recognizing disguised offensive lan- guage, such as adversarial attacks that delib- erately avoid known toxic lexicons, or mani- festations of implicit bias. Building a large annotated dataset for such veiled toxicity can be very expensive. In this work, we propose a framework aimed at fortifying existing toxic speech detectors without a large labeled cor- pus of veiled toxicity. Just a handful of prob- ing examples are used to surface orders of mag- nitude more disguised offenses. We augment the toxic speech detector’s training data with these discovered offensive examples, thereby making it more robust to veiled toxicity while preserving its utility in detecting overt toxic- ity.1 Warning: this paper contains examples that may be offensive or upsetting. 1 Toxic Language in Disguise Toxic language has been recognized as a severe problem in the online social communities. While great efforts have been made to detect and prevent the spread of overt trolling, hate speech, abusive language, and toxic comments (Schmidt and Wie- gand, 2017; Fortuna and Nunes, 2018), they often build upon lexicon-based approaches (Waseem and Hovy, 2016; Davidson et al., 2017) and thus are ineffective at detecting forms of veiled toxicity; e.g., codewords (Taylor et al., 2017), novel forms of offense (Jain et al., 2018), and subtle and often unintentional manifestations of social bias such as microaggressions and condescension (Breitfeller et al., 2019; Wang and Potts, 2019). In this work, we focus on disguised toxic lan- guage that is often undetected by existing tools.2 It 1Our code is available at https://github.com/ xhan77/veiled-toxicity-detection. 2We use the terms veiled and disguised toxicity inter- changeably in this work. can be innocuous on the surface lexicon-level, but as offensive as hate speech and lastingly harmful (Sue et al., 2007; Sue, 2010; Nadal et al., 2014; Jurgens et al., 2019).3 Detecting veiled toxicity is hard: deep seman- tic analysis and large datasets are needed. But veiled offenses are not represented in existing toxi- city datasets (Waseem and Hovy, 2016; Davidson et al., 2017; Founta et al., 2018) and building a new dataset is expensive: candidates for annotation cannot be filtered through lexicons and random sampling of social media posts will surface only a tiny fraction of relevant examples (Breitfeller et al., 2019). Moreover, since biased text is often uncon- scious and subjective, untrained annotators might mislabel it due to their own biases (Breitfeller et al., 2019; Field and Tsvetkov, 2020). We propose a framework to surface veiled of- fenses and improve toxicity classifiers that are com- promised in detecting them. It requires a small set of labeled probing examples to surface orders of magnitude more disguised offenses missed by the classifier, through interpretable ML techniques tracking the influence of training examples on clas- sifier’s …

Content truncated to 500 words