UMUTeam at SemEval-2023 Task 11: Ensemble Learning applied to Binary Supervised Classifiers with disagreements

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1061–1066 July 13-14, 2023 ©2023 Association for Computational Linguistics UMUTeam at SemEval-2023 Task 11: Ensemble Learning applied to Binary Supervised Classifiers with disagreements José Antonio García-Díaz1, Ronghao Pan1, Gema Alcaráz-Mármol2, María José Marín-Pérez3, Rafael Valencia-García1 1 Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100 Murcia, Spain {joseantonio.garcia8, ronghao.pan, valencia}@um.es 2 Departamento de Filología Moderna, Universidad de Castilla La Mancha Facultad de Educación, Avenida Carlos III, 45071 Gema.Alcaraz@uclm.es 3Facultad de Letras, Universidad de Murcia, Campus de la Merced, 30001, Murcia, Spain {angelalm, mariajose.marin1}@um.es Abstract This paper describes the participation of the UMUTeam in the Learning With Disagree- ments (Le-Wi-Di) shared task proposed at Se- mEval 2023, which objective is the develop- ment of supervised automatic classifiers that consider, during training, the agreements and disagreements among the annotators of the datasets. Specifically, this edition includes a multilingual dataset. Our proposal is grounded on the development of ensemble learning classi- fiers that combine the outputs of several Large Language Models. Our proposal ranked posi- tion 18 of a total of 30 participants. However, our proposal did not incorporate the informa- tion about the disagreements. In contrast, we compare the performance of building several classifiers for each dataset separately with a merged dataset. 1 Introduction This manuscript summarizes the participation of the UMUTeam in the second edition of the Learn- ing With Disagreements (Le-Wi-Di) shared task (Leonardellli et al., 2023), proposed in SemEval 2023. Le-Wi-Di focuses on the development of su- pervised automatic classifiers in which the training process considers the agreement and disagreement of the annotators. For this purpose, in this edition of Le-Wi-Di the organizers have released a multilin- gual dataset composed of four normalized datasets focused on entirely subjective tasks. According to the organizers, one of the underlying objectives of this task is to develop a unified testing framework for learning from disagreement. Our proposal to solve the Le-Wi-Di shared task is based on the training of several deep learning classifiers and then combine their outputs using ensemble learning. To this end, we evaluate several Large Language Models (LLMs), and we also con- sider the usage of Linguistic Features (LFs) from UMUTextStats (García-Díaz et al., 2022c). Be- sides, we evaluate the training of a unique classifier for all datasets, or the strategy of training super- vised classifiers specialized in each of the datasets. However, our major limitation is that we could not incorporate to the training of our models the infor- mation about disagreement. Our participation outperformed the baseline and other participants, even though it does not incorpo- rate disagreement. Accordingly, our proposal could be a more real baseline than the one proposed by the organizers of the task. In addition, we have created a GitHub1 reposi- tory with some resources concerning our participa- tion. 2 Background One of the key challenges of this edition of Le- Wi-Di is that it is not focused on a specific topic. Instead of this, the organizers of the task proposed …

Content truncated to 500 words