iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1660–1669 July 13-14, 2023 ©2023 Association for Computational Linguistics iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives? Nikolas Vitsakis1 and Amit Parekh1 and Tanvi Dinkar1 and Gavin Abercrombie1 and Ioannis Konstas1,2 and Verena Rieser1,2 1The Interaction Lab, Heriot-Watt University 2Alana AI nv2006@hw.ac.uk Abstract There are two competing approaches for mod- elling annotator disagreement: distributional soft-labelling approaches (which aim to cap- ture the level of disagreement) or modelling perspectives of individual annotators or groups thereof. We adapt a multi-task architecture — which has previously shown success in mod- elling perspectives— to evaluate its perfor- mance on the SEMEVAL Task 11. We do so by combining both approaches, i.e. pre- dicting individual annotator perspectives as an interim step towards predicting annotator dis- agreement. Despite its previous success, we found that a multi-task approach performed poorly on datasets which contained distinct an- notator opinions, suggesting that this approach may not always be suitable when modelling perspectives. Furthermore, our results explain that while strongly perspectivist approaches might not achieve state-of-the-art performance according to evaluation metrics used by distri- butional approaches, our approach allows for a more nuanced understanding of individual perspectives present in the data. We argue that perspectivist approaches are preferable because they enable decision makers to amplify minor- ity views, and that it is important to re-evaluate metrics to reflect this goal. 1 Introduction Many Natural Language Processing (NLP) tasks follow a supervised learning paradigm, i.e. classifi- cation of labelled data where multiple annotations are aggregated into a hard label using averaging or majority voting. Hard labels are based on the assumption that each instance in a dataset has one singularly correct response—often referred to as ‘ground truth’. However, this assumption is highly unrealistic for social computing tasks, such as such as tox- icity and hate speech detection (e.g. Vidgen and Derczynski, 2021), where lived experiences, bi- ases, opinions and annotator experience all play a role in the subjective response an annotator might give. Hard labels especially disadvantage minority groups (Blodgett, 2021). For example, in abusive language classification, where a minority is dispro- portionately affected (such as minoritised people who have faced online harassment), an aggregated majority label can obscure the perspective of the most vulnerable groups. Thus, there is growing awareness that modelling multiple perspectives is necessary, particularly for inherently subjective tasks and those concerned with social issues (Abercrombie et al., 2022; Cab- itza et al., 2023; Plank, 2022). Le-Wi-Di The SemEval 2023 shared task ‘Learn- ing With Disagreements’ (Leonardellli et al., 2023) aims to capture and model annotator disagreement – going beyond the assumption of one aggregated ‘ground truth’ label. Participating teams are re- quired to propose methods that consider disaggre- gated annotations, used to create a soft label, which represents the probabilistic distribution of the anno- tations. Soft labels can then be used to predict the level of disagreement for each instance in a dataset. The task presents a benchmark of four datasets, including …

Content truncated to 500 words