Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3226–3240 August 1–6, 2021. ©2021 Association for Computational Linguistics 3226 Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech Margherita Fanton1,2, Helena Bonaldi1,2, Serra Sinem Tekiro˘glu2, Marco Guerini2, 1University of Trento, Italy 2Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy mfanton@fbk.eu, hbonaldi@fbk.eu, tekiroglu@fbk.eu, guerini@fbk.eu Abstract Undermining the impact of hateful content with informed and non-aggressive responses, called counter narratives, has emerged as a possible solution for having healthier online communities. Thus, some NLP studies have started addressing the task of counter narra- tive generation. Although such studies have made an effort to build hate speech / counter narrative (HS/CN) datasets for neural gener- ation, they fall short in reaching either high- quality and/or high-quantity. In this paper, we propose a novel human-in-the-loop data collection methodology in which a generative language model is refined iteratively by us- ing its own data from the previous loops to generate new training samples that experts re- view and/or post-edit. Our experiments com- prised several loops including dynamic vari- ations. Results show that the methodology is scalable and facilitates diverse, novel, and cost-effective data collection. To our knowl- edge, the resulting dataset is the only expert- based multi-target HS/CN dataset available to the community. 1 Introduction The proliferation of online hatred has became an alarming issue (Williams, 2019) threatening not only the well-being of target individuals and groups, but also of society as a whole. While authorities establish regulations and policies, so- cial media platforms take actions against hate speech mostly through moderation activities, such as content removal, account suspension, or shadow- banning, at the risk of hindering the freedom of expression. Meanwhile, Non-Governmental Orga- nizations are qualifying volunteers for responding to online hate to promote human dignity and under- standing in society. Such responses, i.e., Counter- Narratives (CN), are non-aggressive textual feed- back using credible evidence, factual arguments, alternative viewpoints, and are considered as an ef- fective strategy (Benesch, 2014; Schieb and Preuss, 2016) to confront hate speech while respecting the human rights (Kiritchenko et al., 2020). However, the vast amount of online hate speech makes an effective manual intervention impossible, which motivates a line of NLP research focusing on semi or fully automatized CN generation so- lutions1. In recent years, several CN collection strategies and datasets have been proposed address- ing the data-hungry nature of current state of the art generation technologies (Mathew et al., 2018; Qian et al., 2019; Chung et al., 2019). Considering the shortcomings of the existing collection strategies (that grant either quality or quantity, but not both), we present an approach to produce high quality CNs for multiple hate targets while reducing the need for expert intervention. To this end, we build on top of the previous hybrid data collection strategies, aiming to increase effi- ciency while maintaining the requirements of data quality, novelty and diversity. In particular, we start …

Content truncated to 500 words