Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1978–1982 July 13-14, 2023 ©2023 Association for Computational Linguistics Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty Ruyuan Wan, Karla Badillo-Urquiola University of Notre Dame rwan@nd.edu, kbadill3@nd.edu Abstract This study investigates learning with disagree- ment in NLP tasks and evaluates its perfor- mance on four datasets. The results suggest that the model performs best on the experimen- tal dataset and faces challenges in minority lan- guages. Furthermore, the analysis indicates that annotator demographics play a significant role in the interpretation of such tasks. This study suggests the need for greater considera- tion of demographic differences in annotators and more comprehensive evaluation metrics for NLP models. 1 Introduction Which voices should an algorithm be trained to replicate? In natural language processing, detect- ing and handling disagreements among annotators is a critical task. Annotators may disagree due to various reasons, including differences in personal beliefs, cultural background, or subjective interpre- tations. Disagreements in annotated data may lead to biased models, which can have negative impli- cations, such as amplifying hate speech or discrim- ination. However, in the current supervised learn- ing paradigm, annotation disagreements are often resolved by aggregating the opinions of majority groups, effectively excluding minority perspectives. Therefore, it is essential to develop methods that can capture and address disagreements in anno- tated data. This paper reports analysis of partici- pating the SemEval-2023 Task 11, Learning with Disagreement, to promote a better understanding of disagreements among annotators and their impact on natural language processing models. 2 Background The use of crowdsourcing for annotating data has become a popular solution for the expensive and time-consuming process of labeling large amounts of data in NLP models. However, it also brings its own set of challenges, with annotator disagree- ment being a primary concern (Checco et al., 2017; Kairam and Heer, 2016). Annotator disagreement can have various causes, but subjective judgment and language ambiguity are common culprits (Uma et al., 2022). Failure to consider the subjective and ambiguous nature of some instances can result in inaccurate predictions. 2.1 Annotation Disagreement Annotation disagreement is a prevalent challenge in NLP datasets, where different annotators label the same instance with different labels. There can be various reasons why annotators disagree, such as the subjective nature of the task, the ambiguity of language, or personal biases of the annotators. Even objective tasks can have multiple acceptable answers, leading to disagreement among annota- tors. Subjectivity is a significant cause of disagree- ment, as tasks like sentiment analysis or hate speech detection require annotators to make subjec- tive judgments. Annotators’ personal experiences, cultural backgrounds, and beliefs can influence their perception of the text, leading to conflicting labels(Wan et al., 2023). Additionally, ambiguity in language can also cause disagreement. Words and phrases can have multiple meanings or interpre- tations, leading to differences in labeling. Further- more, personal biases of annotators can also play a role in annotation disagreement. Annotators may bring their …

Content truncated to 500 words