NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 488–497 July 13-14, 2023 ©2023 Association for Computational Linguistics NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis Mingyang Wang1,2, Heike Adel1, Lukas Lange1, Jannik Strötgen1,3, Hinrich Schütze2 1 Bosch Center for Artificial Intelligence, Renningen, Germany 2 Center for Information and Language Processing (CIS), LMU Munich, Germany 3 Karlsruhe University of Applied Sciences, Karlsruhe, Germany {Mingyang.Wang2,Heike.Adel,Lukas.Lange}@de.bosch.com Abstract This paper describes our system developed for the SemEval-2023 Task 12 “Sentiment Analy- sis for Low-resource African Languages us- ing Twitter Dataset”. Sentiment analysis is one of the most widely studied applications in natural language processing. However, most prior work still focuses on a small number of high-resource languages. Building reliable sen- timent analysis systems for low-resource lan- guages remains challenging, due to the lim- ited training data in this task. In this work, we propose to leverage language-adaptive and task-adaptive pretraining on African texts and study transfer learning with source language selection on top of an African language-centric pretrained language model. Our key findings are: (1) Adapting the pretrained model to the target language and task using a small yet rele- vant corpus improves performance remarkably by more than 10 F1 score points. (2) Selecting source languages with positive transfer gains during training can avoid harmful interference from dissimilar languages, leading to better re- sults in multilingual and cross-lingual settings. In the shared task, our system wins 8 out of 15 tracks and, in particular, performs best in the multilingual evaluation. 1 Introduction In recent years, natural language processing re- search has attracted considerable interest. How- ever, most studies remain confined to a small num- ber of languages with large amounts of training data available. Low-resource languages, for exam- ple, e.g., African languages, are still underrepre- sented although they are spoken by over a billion people. In this context, the AfriSenti shared task provides a Twitter dataset for sentiment analysis on 14 African languages, promoting the future de- velopment of this field. The shared task consists of three sub-tasks: monolingual (Subtask A), mul- tilingual (Subtask B), and zero-shot cross-lingual sentiment analysis (Subtask C). A detailed descrip- tion can be found in the shared task description papers (Muhammad et al., 2023a,b). In this paper, we describe our submission as Nei- ther Language Nor Domain Experts (NLNDE)1 to the AfriSenti shared task. Given the key challenge of limited training data, we first adopt the language- adaptive and task-adaptive pretraining approaches (Gururangan et al., 2020), i.e., LAPT and TAPT, to adapt a pretrained language model to the language and task of interest. Further pretraining the model with such smaller but more task-relevant corpora leads to performance gains in all subtasks. Second, Cross-lingual transfer has been shown to be an effective method for enhancing the per- formance of low-resource languages by leveraging one or more similar languages as source languages (Lin et al., 2019; Ruder and Plank, 2017; Nasir and Mchechesi, 2022). However, the 14 African languages …

Content truncated to 500 words