Annotating for Hate Speech: The MaNeCo Corpus and Some Input from Critical Discourse Analysis

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5088–5097 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5088 Annotating for Hate Speech: The MaNeCo Corpus and Some Input from Critical Discourse Analysis Stavros Assimakopoulos, Rebecca Vella Muskat, Lonneke van der Plas, Albert Gatt University of Malta Institute of Linguistics and Language Technology, Msida MSD 2080, Malta {stavros.assimakopoulos, rebecca.vella, lonneke.vanderplas, albert.gatt}@um.edu.mt Abstract This paper presents a novel scheme for the annotation of hate speech in corpora of Web 2.0 commentary. The proposed scheme is motivated by the critical analysis of posts made in reaction to news reports on the Mediterranean migration crisis and LGBTIQ+ matters in Malta, which was conducted under the auspices of the EU-funded C.O.N.T.A.C.T. project. Based on the realization that hate speech is not a clear-cut category to begin with, appears to belong to a continuum of discriminatory discourse and is often realized through the use of indirect linguistic means, it is argued that annotation schemes for its detection should refrain from directly including the label ’hate speech,’ as different annotators might have different thresholds as to what constitutes hate speech and what not. In view of this, we suggest a multi-layer annotation scheme, which is pilot-tested against a binary ±hate speech classification and appears to yield higher inter-annotator agreement. Motivating the postulation of our scheme, we then present the MaNeCo corpus on which it will eventually be used; a substantial corpus of on-line newspaper comments spanning 10 years. Keywords: hate speech, annotation, newspaper comments, corpus creation 1. Introduction While the discussion of automatic hate speech detection goes back at least two decades (Spertus, 1997; Greevy and Smeaton, 2004), recent years have witnessed a renewed interest in the area. This is largely due to the prolifera- tion of user-generated content as part of Web 2.0, which has given rise to continuous streams of content, produced in large volumes over multiple geographical regions and in multiple languages and language varieties. As a result, the exponential increase in potential sources of hate speech in combination with the continuous introduction of legisla- tion and policy-making aiming specifically at regulating the phenomenon across a number of countries (Banks, 2010) clearly necessitates the development of reliable automatic methods for detecting hate speech online that will comple- ment human moderation. From a natural language processing (NLP) perspective, treatments of hate speech focus mainly on the problem of identification (Fortuna and Nunes, 2018). Thus, given a span of text, the task is to identify whether it is an in- stance of hate speech or not. This makes the problem a case of binary classification (±hate speech), which in turn makes it amenable to treatment using a variety of classifica- tion methods. These supervised learning techniques require pre-labelled training data, consisting of manually annotated positive and negative examples of the class(es) to be iden- tified, to learn a model which, given a new instance, can predict the label with some degree of probability. In this setting, …

Content truncated to 500 words