Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1667–1682 August 1–6, 2021. ©2021 Association for Computational Linguistics 1667 Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection Bertie Vidgen†, Tristan Thrush‡, Zeerak Waseem⋆, Douwe Kiela‡ †The Alan Turing Institute; ⋆University of Sheffield; ‡Facebook AI Research bvidgen@turing.ac.uk Abstract We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more ro- bust hate detection models. We provide a new dataset of ∼40, 000 entries, generated and la- belled by trained annotators over four rounds of dynamic data creation. It includes ∼15, 000 challenging perturbations and each hateful en- try has fine-grained labels for the type and tar- get of hate. Hateful entries make up 54% of the dataset, which is substantially higher than comparable datasets. We show that model per- formance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also have better performance on HATECHECK, a suite of functional tests for online hate detec- tion. We provide the code, dataset and annota- tion guidelines for other researchers to use. 1 Introduction Accurate detection of online hate speech is impor- tant for ensuring that such content can be found and tackled scalably, minimizing the risk that harm will be inflicted on victims and making online spaces more accessible and safe. However, detecting on- line hate has proven remarkably difficult and con- cerns have been raised about the performance, ro- bustness, generalisability and fairness of even state- of-the-art models (Waseem et al., 2018; Vidgen et al., 2019a; Caselli et al., 2020; Mishra et al., 2019; Poletto et al., 2020). To address these chal- lenges, we present a human-and-model-in-the-loop process for collecting data and training hate detec- tion models. Our approach encompasses four rounds of data generation and model training. We first trained a classification model using previously released hate speech datasets. We then tasked annotators with presenting content that would trick the model and yield misclassifications. At the end of the round we trained a new model using the newly presented data. In the next round the process was repeated with the new model in the loop for the annotators to trick. We had four rounds but this approach could, in principle, be continued indefinitely. Round 1 contains original content created syn- thetically by annotators. Rounds 2, 3 and 4 are split into half original content and half pertur- bations. The perturbations are challenging ‘con- trast sets’, which manipulate the original text just enough to flip the label (e.g. from ‘Hate’ to ‘Not Hate’) (Kaushik et al., 2019; Gardner et al., 2020). In Rounds 3 and 4 we also tasked annotators with exploring specific types of hate and taking close in- spiration from real-world hate sites to make content as adversarial, realistic, and varied as possible. Models have lower accuracy when evaluated on …

Content truncated to 500 words