SemEval-2021 Task 12: Learning with Disagreements

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338–347 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 338 SemEval-2021 Task 12: Learning with Disagreements Alexandra Uma1 Tommaso Fornaciari2 Anca Dumitrache3 Tristan Miller4 Jon Chamberlain5 Barbara Plank6 Edwin Simpson7 Massimo Poesio1 1Queen Mary University of London 2Università Bocconi, Milano 3Albert Heijn 4Austrian Research Institute for Artificial Intelligence 5University of Essex 6IT University of Copenhagen 7University of Bristol m.poesio@qmul.ac.uk Abstract Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language process- ing and computer vision. However, most su- pervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on Learning with Disagreements (Le-wi-Di) was to provide a unified testing framework for meth- ods for learning from data containing multiple and possibly contradictory annotations cover- ing the best-known datasets containing infor- mation about disagreements for interpreting language and classifying images. In this pa- per we describe the shared task and its results. 1 Introduction The assumption that natural language (nl) expres- sions have a single and clearly identifiable inter- pretation in a given context, or that images have a preferred labels, still underlies most work in nlp and computer vision. However, there is now plenty of evidence that this assumption is just a conve- nient idealization; virtually every project devoted to large-scale annotation has found that genuine disagreements are widespread. In nlp, that annotator/coder disagreement can be genuine—i.e., resulting from debatable, diffi- cult, or linguistic ambiguity—has long been known for anaphora and coreference (Poesio and Artstein, 2005; Versley, 2008; Recasens et al., 2011).1 But in recent years, we have also seen evidence that disagreements among subjects/coders are common with virtually every aspect of language interpreta- tion, from apparently simple aspects such as part- of-speech tagging (Plank et al., 2014b), to more 1See also the analysis of disagreements in OntoNotes and word senses in Pradhan et al. (2012), Passonneau et al. (2012), and Martínez Alonso et al. (2016). complex ones like semantic role assignment (Du- mitrache et al., 2019), to subjective tasks such as sentiment analysis (Kenyon-Dean et al., 2018), and to the inferences that can be drawn from sen- tences (Pavlick and Kwiatkowski, 2019). In computer vision, as well, the assumption that gold labels may be specified for items has proven an idealization (Rodrigues and Pereira, 2018)—in fact, possibly even more than for nlp. In many widely used crowdsourced datasets for computer vi- sion, different coders assign equally plausible labels to the same items. The problem of disagreement among coders, including experts, on the classifi- cation of noisy image data has arisen in many cv applications. This includes classification of astro- nomical images (Smyth et al., 1994), medical image classification (Raykar et al., 2010), and numerous others (Sharmanska et al., 2016; Rodrigues and Pereira, 2018; Firman et al., 2018). Many ai researchers have concluded that rather than attempting to eliminate disagreements from an- …

Content truncated to 500 words