MIND at SemEval-2023 Task 11: From Uncertain Predictions to Subjective Disagreement

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 556–564 July 13-14, 2023 ©2023 Association for Computational Linguistics MIND at SemEval-2023 Task 11: From Uncertain Predictions to Subjective Disagreement Giulia Rizzi1,2, Alessandro Astorino1, Daniel Scalena1, Paolo Rosso2, Elisabetta Fersini1 1University of Milano-Bicocca, Viale Sarca, 336 - Milan, Italy 2Universitat Politècnica de València, Camino de Vera, Valencia, Spain {g.rizzi10, a.astorino2, d.scalena} @campus.unimib.it prosso@dsic.upv.es, elisabetta.fersini@unimib.it Abstract This paper describes the participation of the research laboratory MIND, at the University of Milano-Bicocca, in the SemEval 2023 task related to Learning With Disagreements (Le- Wi-Di). The main goal is to identify the level of agreement/disagreement from a collection of textual datasets with different characteris- tics in terms of style, language, and task. The proposed approach is grounded on the hypoth- esis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic character- istics, could have on the prediction of a given gold label. 1 Introduction Nowadays, natural language processing models are consistently adopted to address several tasks, such as Abusive and Offensive Language Detection (Pradhan et al., 2020; Kaur et al., 2021) and Hate Speech Detection (Alkomah and Ma, 2022). The state-of-the-art works address both general and spe- cific targets (e.g., women, LGBTQ, immigrants) from a unimodal and multimodal perspective, also considering several languages ((Fersini et al., 2022; Magnossao de Paula et al., 2021; Chakravarthi et al., 2022; Lee et al., 2022; Muaad et al., 2022). However, although these tasks are subjective, the majority of the proposed works are based on the assumption that a unique perception and interpreta- tion exists for each instance. However, this assump- tion does not reflect what happens in the real world, where multiple readers could have different points of view and understanding, resulting in a sort of disagreement. Such disagreement, which typically reflects the subjectivity of the task, can be due to linguistic ambiguity or different beliefs originated from different socio-cultural aspects. To this purpose, many researchers have proposed corpora that aim at capturing all the distinctive interpretations of a given text by preserving the annotations disagreement instead of just aggregat- ing them in a single gold label. The informa- tion brought by the annotators’ disagreement has been mainly exploited in three different ways: (1) to improve the quality of the dataset by remov- ing those instances characterized by a disagree- ment between annotators (Beigman Klebanov and Beigman, 2009), (2) to weight the instances dur- ing the training phase (Dumitrache et al., 2019) or (3) to directly train a machine learning model from disagreement, without considering any aggre- gated label (Uma et al., 2021b; Fornaciari et al., 2021). An important contribution in this field has been given by the SemEval-2021 Task 12 - Learn- ing with Disagreements (Le-Wi-Di) (Uma et al., 2021a), whose aim was to provide a unified test- ing framework from disagreement for interpreting language and classifying images. In this paper, we address the Learning with Dis- agreements (Le-Wi-Di) task at SemEval-2023 Task 11 (Leonardellli …

Content truncated to 500 words