Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8031 - 8049 December 7-11, 2022 ©2022 Association for Computational Linguistics Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering Helena Bonaldi1,3, Sara Dellantonio2,3, Serra Sinem Tekiro˘glu3, Marco Guerini3, 1University of Trento, Italy 2Free University of Bozen-Bolzano, Italy 3Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy hbonaldi@fbk.eu, sdellantonio@fbk.eu, tekiroglu@fbk.eu, guerini@fbk.eu Abstract Fighting online hate speech is a challenge that is usually addressed using Natural Language Processing via automatic detection and removal of hate content. Besides this approach, counter narratives have emerged as an effective tool em- ployed by NGOs to respond to online hate on social media platforms. For this reason, Nat- ural Language Generation is currently being studied as a way to automatize counter narra- tive writing. However, the existing resources necessary to train NLG models are limited to 2-turn interactions (a hate speech and a counter narrative as response), while in real life, inter- actions can consist of multiple turns. In this pa- per, we present a hybrid approach for dialogical data collection, which combines the interven- tion of human expert annotators over machine generated dialogues obtained using 19 different configurations. The result of this work is DI- ALOCONAN, the first dataset comprising over 3000 fictitious multi-turn dialogues between a hater and an NGO operator, covering 6 targets of hate. 1 Introduction While hate towards vulnerable groups or individu- als is not a new phenomenon, the upsurge of hate speech and its proliferation is relatively recent and it is enabled by the fast spread of information in online platforms. The rise in hate speech online can even provoke violent actions offline. Consequently, fighting online Hate Speech (HS) has become a vitally important “job for everyone”1 especially for the NLP researchers. The contrast to HS and haters on social media platforms is usually carried on via user suspension, content removal or shadow ban- ning, which can be mapped to a classification task in NLP terms. However, AI and NLP can play even a more crucial role that is not limited to classifica- tion. In fact, recently, NLG models have started to 1“Hatred is a danger to everyone – and so fighting it must be a job for everyone.” António Guterres, United Nations Secretary-General, 2021 Figure 1: Exemplar dialogue between a hater and an NGO operator. be proposed as an effective tool to counter HS by providing relevant responses. In particular, the idea is to imitate the operators of Non-Governmental Organizations (NGO) that are actually intervening in online discussions by replying to hateful content using so-called Counter Narratives (CN), defined by Schieb and Preuss (2016) as “communicative ac- tions aimed at refuting hate speech through thought- ful and cogent reasons, and true and fact-bound ar- guments”. Through automatically generating CNs, it is possible to aid NGO operators in their day- to-day manual activities, and therefore to partially countervail the sheer amount of hateful content posted online (Chung et al., 2021b). Despite the invaluable …

Content truncated to 500 words