UMUTeam and SINAI at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis using Multilingual Large Language Models and Data Augmentation

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 293–299 July 13-14, 2023 ©2023 Association for Computational Linguistics UMUTeam and SINAI at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis using Multilingual Large Language Models and Data Augmentation José Antonio García-Díaz1, Ronghao Pan1, Salud María Jiménez Zafra2, María-Teresa Martín-Valdivia2, L. Alfonso Ureña-López2, Rafael Valencia-García1 1 Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100 Murcia, Spain {joseantonio.garcia8, ronghao.pan, valencia}@um.es 2Computer Science Department, SINAI, CEATIC, Universidad de Jaén, 23071, Spain {sjzafra, maite, laurena}@ujaen.es Abstract This work presents the participation of the UMUTeam and the SINAI research groups in the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis. The goal of this task is to predict the intimacy of a set of tweets in 10 lan- guages: English, Spanish, Italian, Portuguese, French, Chinese, Hindi, Arabic, Dutch and Ko- rean, of which, the last 4 are not in the training data. Our approach to address this task is based on data augmentation and the use of three mul- tilingual Large Language Models (multilingual BERT, XLM and mDeBERTA) by ensemble learning. Our team ranked 30th out of 45 partic- ipants. Our best results were achieved with two unseen languages: Korean (16th) and Hindi (19th). 1 Introduction In Natural Language Processing (NLP), intimacy can be described as how people communicate their perception and willingness to share personal data and emotions to their audience (Pei and Jurgens, 2020). The SemEval 2023 Task 9, entitled Multilin- gual Tweet Intimacy Analysis (MTIA) (Pei et al., 2023), consists of a regression task in which the participants should rate in a score from 1 to 5 the in- timacy of short documents written in 10 languages: English, Spanish, Italian, Portuguese, French, Chi- nese, Hindi, Arabic, Dutch and Korean. This task was co-organized by University of Michigan and Snap Inc. There are two main challenges concern- ing this task. On the one hand, the training dataset provided to the participants does not cover all the evaluated languages, but only six of them: English, Spanish, Italian, Portuguese, French, and Chinese. However, the evaluation is conducted in those six languages plus Hindi, Arabic, Dutch and Korean. On the other hand, participants were only allowed to submit a unique run, which hinders the shared task. Our strategy to solve the MTIA challenge con- sists of an ensemble learning composed of three multilingual Large Language Models (LLM): mul- tilingual BERT (Devlin et al., 2018), XLM (Lample and Conneau, 2019), and mDeBERTA (He et al., 2021). Besides, we use data augmentation incor- porating to the training the dataset suggested by the organizers and provided in the work of Pei and Jurgens (2020), with more than two thousand En- glish questions from Reddit and other sources and annotated with intimacy scores in the range [-1, 1]. Our participation achieved modest results in the task, reaching the 30th position in the leader-board, with a Pearson’s R of 0.53. The best result is achieved by Lazybob, with a Pearson’s R of 0.62. As commented above, as the …

Content truncated to 500 words