A little goes a long way: Improving toxic language classification despite data scarcity

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2991–3009 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2991 A little goes a long way: Improving toxic language classification despite data scarcity Mika Juuti1, Tommi Gr¨ondahl2, Adrian Flanagan3, N. Asokan1,2 University of Waterloo1 Aalto University2 Huawei Technologies Oy (Finland) Co Ltd3 mika.juuti@kela.fi, tommi.grondahl@aalto.fi adrian.flanagan@huawei.com, asokan@acm.org Abstract Detection of some types of toxic language is hampered by extreme scarcity of labeled train- ing data. Data augmentation – generating new synthetic data from a labeled seed dataset – can help. The efficacy of data augmentation on toxic language classification has not been fully explored. We present the first systematic study on how data augmentation techniques impact performance across toxic language classifiers, ranging from shallow logistic regression ar- chitectures to BERT – a state-of-the-art pre- trained Transformer network. We compare the performance of eight techniques on very scarce seed datasets. We show that while BERT performed the best, shallow classifiers performed comparably when trained on data augmented with a combination of three tech- niques, including GPT-2-generated sentences. We discuss the interplay of performance and computational overhead, which can inform the choice of techniques under different con- straints. 1 Introduction Toxic language is an increasingly urgent challenge in online communities (Mathew et al., 2019). Al- though there are several datasets, most commonly from Twitter or forum discussions (Badjatiya et al., 2017; Davidson et al., 2017; Waseem and Hovy, 2016; Wulczyn et al., 2017; Zhang et al., 2018), high class imbalance is a problem with certain classes of toxic language (Breitfeller et al., 2019). Manual labeling of toxic content is onerous, haz- ardous (Newton, 2020), and thus expensive. One strategy for mitigating these problems is data augmentation (Wang and Yang, 2015; Rat- ner et al., 2017; Wei and Zou, 2019): comple- menting the manually labeled seed data with new synthetic documents. The effectiveness of data augmentation for toxic language classification has not yet been thoroughly explored. On relatively small toxic language datasets, shallow classifiers have been shown to perform well (Gr¨ondahl et al., 2018). At the same time, pre-trained Transformer networks (Vaswani et al., 2017) have led to im- pressive results in several NLP tasks (Young et al., 2018). Comparing the effects of data augmentation between shallow classifiers and pre-trained Trans- formers is thus of particular interest. We systematically compared eight augmentation techniques on four classifiers, ranging from shal- low architectures to BERT (Devlin et al., 2019), a popular pre-trained Transformer network. We used downsampled variants of the Kaggle Toxic Comment Classification Challenge dataset (Jigsaw 2018; §3) as our seed dataset. We focused on the threat class, but also replicated our results on another toxic class (§4.6). With some classifiers, we reached the same F1-score as when training on the original dataset, which is 20x larger. However, performance varied markedly between classifiers. We obtained the highest overall results with BERT, increasing the F1-score up to 21% com- pared to training on seed data alone. However, augmentation using a fine-tuned GPT-2 (§3.2.4) – …

Content truncated to 500 words