CICL_DMS at SemEval-2023 Task 11: Learning With Disagreements (Le-Wi-Di)

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1030–1036 July 13-14, 2023 ©2023 Association for Computational Linguistics CICL_DMS at SemEval-2023 Task 11: Learning With Disagreements (Le-Wi-Di) Dennis Grötzinger and Simon Heuschkel and Matthias Drews University of Tübingen, Tübingen, Germany {firstName.lastName}@student.uni-tuebingen.de Abstract In this system paper, we describe our submis- sion for the 11th task of SemEval2023: Learn- ing with Disagreements, or Le-Wi-Di for short. In the task, the assumption that there is a single gold label in NLP tasks such as hate speech or misogyny detection is challenged, and in- stead the opinions of multiple annotators are considered. The goal is instead to capture the agreements/disagreements of the annotators. For our system, we utilize the capabilities of modern large-language models as our backbone and investigate various techniques built on top, such as ensemble learning, multi-task learn- ing, or Gaussian processes. Our final submis- sion shows promising results and we achieve an upper-half finish. 1 Introduction This paper is a description of the methods we used for our entry1 on this year’s SemEval Task 11, Learning with Disagreements (Le-Wi-Di) compe- tition (Leonardellli et al., 2023). SemEval is a yearly workshop which comprises various natural language processing shared tasks. Each team that participates in one of these tasks will try to come up with systems that deepen the understanding or improve results on one kind of semantic evalua- tion challenge. The task we participated in was task 11, Learning with Disagreements (Le-Wi-DI). The goal of the task was to investigate how learn- ing can be best achieved when there is no single gold label available, and instead the opinions of multiple annotators are given. For instance, con- sider the case of trying to classify if a sentence x contains hate speech or not. Instead of pretending that there is a single ground-truth, binary label to it that every human would agree on, in the task we were provided with a multitude of labels y for x (i.e. y = [0,1,0,0]), where each label reflects the 1Available at https://github.com/cicl-iscl/ LeWiDi_SemEval2023 opinion of one annotator. Thus, there were both more uncontroversial instances where most annota- tors agreed and also more controversial instances where annotators disagreed a lot. Scoring was done in two ways: First, standard F1 scores were calculated ("hard evaluation"), based upon a majority voting of the annotator’s opinion as labels. However, as this was exactly the assump- tion that the task tries to overcome, hard evaluation was only secondary and mainly used to provide a link to other research. Instead, the main eval- uation criterion was "soft evaluation". Here, the cross-entropy between the predictions and the dis- aggregated crowd-annotations label is measured. The disaggregated crowd-annotations label is de- scribing how many annotators labeled with 1 rel- ative to 0. For instance, the disaggregated crowd- annotations label for y=[0,0,0,0] would be [1,0] and for y=[1,0,0,1] it would be [0.5, 0.5]. Four datasets were included in the task. Each of them had the individual annotator labels given and also the …

Content truncated to 500 words