SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2304–2318 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 11: Learning With Disagreements (LeWiDi) Elisa Leonardelli1 Gavin Abercrombie2 Dina Almanea3 Valerio Basile 4 Tommaso Fornaciari5 Barbara Plank 6 Verena Rieser2 Alexandra Uma Massimo Poesio3 1Fondazione Bruno Kessler, 2Heriot-Watt University, 3Queen Mary University of London, 4Università di Torino, 5Università Bocconi, 6University of Munich Abstract � The paper contains examples which are offensive in nature. nlp datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the nlp community has come to realize that the approach of ‘reconcil- ing’ these different subjective interpretations is inappropriate. Many nlp researchers have therefore concluded that rather than eliminat- ing disagreements from annotated corpora, we should preserve them–indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus cre- ation for nlp has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing nlp models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on nlp, instead of both nlp and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements–as training with aggregated la- bels for subjective nlp tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft ap- proaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers. 1 Introduction In recent years, the assumption that natural lan- guage (nl) expressions have a single and clearly identifiable interpretation in a given context is in- creasingly recognized as just a convenient idealiza- tion. Virtually every large-scale annotation project in the last twenty years has found widespread evi- dence of disagreement (Poesio and Artstein, 2005; Versley, 2008; Recasens et al., 2011; Passonneau et al., 2012; Plank et al., 2014b; Martínez Alonso et al., 2016; Dumitrache et al., 2019). Plenty of evidence has also been found of disagreements over the inferences that underlie much of semantic inter- pretation (Pavlick and Kwiatkowski, 2019). More recently, the increasing focus in nlp on tasks de- pending on subjective judgments such as sentiment analysis (Kenyon-Dean et al., 2018), humour and sarcasm identification (Simpson et al., 2019) and offensive language detection (Cercas Curry et al., 2021; Leonardelli et al., 2021; Akhtar et al., 2021) led to the realization that in such tasks ‘reconciling’ different subjective interpretations does not make much sense (Basile, 2020; Basile et al., 2021). Many ai researchers concluded therefore that rather than eliminating disagreements from anno- tated corpora, we should preserve them–indeed, some researchers have argued that corpora should …

Content truncated to 500 words