Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528–10539 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10528 Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio Marco Guerini, Sara Tonelli Fondazione Bruno Kessler, Trento, Italy {eleonardelli,menini,aprosio,guerini,satonelli}@fbk.eu Abstract Since state-of-the-art approaches to offensive language detection rely on supervised learn- ing, it is crucial to quickly adapt them to the continuously evolving scenario of social me- dia. While several approaches have been pro- posed to tackle the problem from an algorith- mic perspective, so to reduce the need for an- notated data, less attention has been paid to the quality of these data. Following a trend that has emerged recently, we focus on the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. Our study comprises the creation of three novel datasets of English tweets covering different topics and having five crowd-sourced judg- ments each. We also present an extensive set of experiments showing that selecting training and test data according to different levels of an- notators’ agreement has a strong effect on clas- sifiers performance and robustness. Our find- ings are further validated in cross-domain ex- periments and studied using a popular bench- mark dataset. We show that such hard cases, where low agreement is present, are not nec- essarily due to poor-quality annotation and we advocate for a higher presence of ambiguous cases in future datasets, particularly in test sets, to better account for the different points of view expressed online. 1 Introduction When creating benchmarks for NLP tasks through crowd-sourcing platforms, it is important to con- sider possible issues with inter-annotator agree- ment. Indeed, crowd-workers do not necessarily have a linguistic background and are not trained to perform complex tasks, thus jeopardizing bench- mark quality. Furthermore, some crowd-workers try to maximize their pay by supplying quick an- swers that have nothing to do with the correct label. This issue has been tackled in the past by propos- ing approaches to control for annotators’ expertise and reliability (Hovy et al., 2013), trying to iden- tify spammers and mitigate their effect on annota- tion, or by repeating labeling on targeted examples (Sheng et al., 2008). However, not all tasks are the same: while in some cases, like for instance PoS- tagging or parsing, disagreement among annotators is more likely due to unclear annotation guidelines and can usually be reconciled through adjudication, full annotators’ agreement should not be necessar- ily enforced in social computing tasks, whose goal is to study and manage social behavior and orga- nizational dynamics, especially in virtual worlds built over the Internet (Wang, 2007). In these tasks – which include offensive language detection among others – subjectivity, bias and text ambiguity play an important role (Aroyo et al., 2019), and being an inherent component of the task they should be measured and analysed rather than discarded (Klen- ner et al., 2020; Basile, …

Content truncated to 500 words