Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2428–2441 May 2-6, 2023 ©2023 Association for Computational Linguistics Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks Marta Sandri Dept. of Humanities University of Pavia, Italy sandri.marta97@gmail.com Elisa Leonardelli Fondazione Bruno Kessler Trento, Italy eleonardelli@fbk.eu Sara Tonelli Fondazione Bruno Kessler Trento, Italy satonelli@fbk.eu Elisabetta Jezek Dept. of Humanities University of Pavia, Italy jezek@unipv.it Abstract Annotators’ disagreement in linguistic data has been recently the focus of multiple initiatives aimed at raising awareness on issues related to ‘majority voting’ when aggregating diverging annotations. Disagreement can indeed reflect different aspects of linguistic annotation, from annotators’ subjectivity to sloppiness or lack of enough context to interpret a text. In this work we first propose a taxonomy of pos- sible reasons leading to annotators’ disagree- ment in subjective tasks. Then, we manually label part of a Twitter dataset for offensive lan- guage detection in English following this tax- onomy, identifying how the different categories are distributed. Finally we run a set of exper- iments aimed at assessing the impact of the different types of disagreement on classifica- tion performance. In particular, we investigate how accurately tweets belonging to different categories of disagreement can be classified as offensive or not, and how injecting data with different types of disagreement in the training set affects performance. We also perform offen- sive language detection as a multi-task frame- work, using disagreement classification as an auxiliary task. Warning: This paper contains examples that may be offensive or upsetting. 1 Introduction The development of benchmark datasets based on the reconciliation of annotators’ disagreement has been a standard practice within the NLP commu- nity for decades. However, in the last few years this paradigm has been questioned (Basile, 2020; Aber- crombie et al., 2022), since aggregating discording labels is based upon the assumption that texts have a single interpretation and that annotators’ disagree- ment is something that should be corrected. On the contrary, it may convey useful information on the task, the data and the annotators themselves (Aroyo and Welty, 2015). For example, annotators may disagree because of poorly described guidelines, or because they interpret in different ways a text based on their background and beliefs. The text may be ambiguous because of a lack of context, or annotators may simply be working poorly, with- out paying much attention to the task they should perform on the text (Uma et al., 2021). Having a better understanding of the reasons be- hind annotators’ disagreement could lead to better annotation guidelines and provide insights into the advantages and limits of developing NLP systems able to deal with disagreement (Davani et al., 2022). It would also contribute to a better understanding of how disagreement interferes with systems’ perfor- mance. Finally, automatically detecting instances that are likely to obtain discording labels could help researchers in creating linguistic datasets with more or less challenging examples (Lehmann et al., 1996). In this work, we contribute to the …

Content truncated to 500 words