xiacui at SemEval-2023 Task 11: Learning a Model in Mixed-Annotator Datasets Using Annotator Ranking Scores as Training Weights

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1076–1084 July 13-14, 2023 ©2023 Association for Computational Linguistics xiacui at SemEval-2023 Task 11: Learning a Model in Mixed-Annotator Datasets using Annotator Ranking Scores as Training Weights Xia Cui Manchester Metropolitan University x.cui@mmu.ac.uk Abstract This paper describes the development of a system for SemEval-2023 Shared Task 11 on Learning with Disagreements (Le-Wi- Di) (Leonardellli et al., 2023). Labelled data plays a vital role in the development of machine learning systems. The human-annotated labels are usually considered the truth for training or validation. To obtain truth labels, a traditional way is to hire domain experts to perform an ex- pensive annotation process. Crowd-sourcing la- belling is comparably cheap, whereas it raises a question on the reliability of annotators. A com- mon strategy in a mixed-annotator dataset with various sets of annotators for each instance is to aggregate the labels among multiple groups of annotators to obtain the truth labels. How- ever, these annotators might not reach an agree- ment, and there is no guarantee of the reliability of these labels either. With further problems caused by human label variation, subjective tasks usually suffer from the different opinions provided by the annotators. In this paper, we propose two simple heuristic functions to com- pute the annotator ranking scores, namely An- noHard and AnnoSoft, based on the hard labels (i.e., aggregative labels) and soft labels (i.e., cross-entropy values). By introducing these scores, we adjust the weights of the training instances to improve the learning with disagree- ments among the annotators. 1 Introduction Annotated datasets are fundamental for any ma- chine learning model. Traditional supervised ma- chine learning models are heavily based on well- labelled datasets. Fine-tuning and validation could not exist without them either. The cost of manual labelling using domain experts is generally high to ensure the annotation quality. With the recent popu- larity of crowd-sourcing, the cost has become lower. However, the reliability of annotators remains a question. Even with domain experts, disagreements are commonly observed in the multiple annotations for the same task due to several circumstances. For example, the task is complex with many factors. It may require annotators’ expertise, such as predict- ing mental health issues (Thieme et al., 2020), or subjective such as detecting emotions or other po- larised opinions (Schuff et al., 2017; Akhtar et al., 2020). The model trained with unreliable labels can potentially increase the cost by human post- validation. In this paper, we focus on a specific sub-problem of learning with disagreements in mixed sets of an- notators from different backgrounds, and it could be raised by crowd-sourcing or a mixture of the crowd and expert labels. We propose a training strategy to weigh the instance by computing the ranking score of the annotator. More specifically, our method is proposed to trust more instances with higher annotator ranks during the training stage. Hence, we propose two functions to compute anno- tator ranking scores: AnnoHard and AnnoSoft. As suggested in the naming, …

Content truncated to 500 words