UMUTeam at SemEval-2023 Task 12: Ensemble Learning of LLMs applied to Sentiment Analysis for Low-resource African Languages

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 285–292 July 13-14, 2023 ©2023 Association for Computational Linguistics UMUTeam at SemEval-2023 Task 12: Ensemble Learning of LLMs applied to Sentiment Analysis for Low-resource African Languages José Antonio García-Díaz1, Camilo Caparros-Laiz1, Ángela Almela2, Gema Alcaráz-Mármol3, María José Marín-Pérez2, Rafael Valencia-García1 1 Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100 Murcia, Spain {joseantonio.garcia8, camilo.caparrosl, valencia}@um.es 2 Facultad de Letras, Universidad de Murcia, Campus de la Merced, 30001 Murcia, Spain {angelalm, mariajose.marin1}@um.es 3 Facultad de Educación, Universidad de Castilla La Mancha, 45071 Toledo, Spain Gema.Alcaraz@uclm.es Abstract These working notes summarize the participa- tion of the UMUTeam in the SemEval 2023 shared task: AfriSenti, focused on Sentiment Analysis in several African languages. Two subtasks are proposed, one in which each lan- guage is considered separately and another one in which all languages are merged. Our pro- posal to solve both subtasks is grounded on the combination of features extracted from sev- eral multilingual Large Language Models and a subset of language-independent linguistic fea- tures. Our best results are achieved with the African languages less represented in the train- ing set: Xitsonga, a Mozambique dialect, with a weighted f1-score of 54.89%; Algerian Arabic, with a weighted f1-score of 68.52%; Swahili, with a weighted f1-score of 60.52%; and Twi, with a weighted f1-score of 71.14%. 1 Introduction The underlying objective of AfriSenti shared task (Muhammad et al., 2023b) is to promote meth- ods, tools, and curated datasets for African lan- guages (Muhammad et al., 2023a). Specifically, the organizers of this task propose a Sentiment Analysis (SA) task focused on 16 African lan- guages, including several languages from Nige- ria, Ethiopia, Kenya, Tanzania, Algeria, Rwanda, Ghana, Mozambique or South Africa. This task is divided into two subtasks. The first one is a monolingual sentiment classification, in which the polarity of a sentence should be determined. In this task, the documents from different languages are not mixed. The second subtask is a multilingual sentiment classification. In both cases, polarity is a multi-classification problem of three labels, namely, positive, neutral and negative. To solve these subtasks, our research group pro- poses a deep-learning classifier based on ensemble learning. This ensemble combines the predictions of a subset of language-independent linguistic fea- tures with features from several Large Language Models (LLMs). The evaluated LLMs include mul- tilingual general purpose models such as BERT, XLM, and DeBERTA; DziriBERT, focused on the Algerian dialect, and AfriBERTa, a multi-lingual LLM trained on 11 languages from Africa. We also included the original BERT as some of the documents in the dataset are written in English. For subtask 1, all languages are evaluated sepa- rately. However, we decided to train all languages together in order to add some instances of underrep- resented languages in the provided corpora. More- over, the majority of the evaluated LLMs are multi- lingual so they can handle documents from differ- ent tracks in the same batch. This idea has worked well, as our best results are …

Content truncated to 500 words