DuluthNLP at SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1697–1701 July 13-14, 2023 ©2023 Association for Computational Linguistics DuluthNLP at SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twitter Dataset Samuel Akrah & Ted Pedersen Department of Computer Science University of Minnesota Duluth, MN 55812 USA {akrah001, tpederse}@d.umn.edu Abstract This paper describes the DuluthNLP system that participated in Task 12 of SemEval-2023 on AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages using Twit- ter Dataset. Given a set of tweets, the task requires participating systems to classify each tweet as negative, positive or neutral. We eval- uate a range of monolingual and multilingual pre-trained models on the Twi language dataset, one among the 14 African languages included in the SemEval task. We introduce TwiBERT, a new pretrained model trained from scratch. We show that TwiBERT, along with mBERT, generally perform best when trained on the Twi dataset, achieving an F1 score of 64.29% on the official evaluation test data, which ranks 14 out of 30 of the total submissions for Track 10. The TwiBERT model is released at https://huggingface.co/sakrah/TwiBERT 1 Introduction Sentiment analysis is a well-studied problem in Natural Language Processing (NLP), at least with respect to high-resourced languages such as En- glish. However, despite the popularity of sentiment analysis it is not well studied in languages where there are limited resources (Joshi et al., 2020). This is particularly true when discussing the languages of Africa. Africa is the most linguistically diverse conti- nent, and is home to more than 2,000 languages. Yet the continent is chronically underserved in NLP resources and technology, in part because there is little to no training data available for African lan- guages in NLP research. However, more recently there have been con- certed efforts to create datasets and build mod- els for African languages (Adelani et al., 2021; Muhammad et al., 2022). One such effort is SemEval-2023 Task 12 (Muhammad et al., 2023b), a sentiment analysis task for low-resource African Languages. This paper describes the DuluthNLP team’s participation in Track 10 of this task which focused on the Twi language. 2 Task Data The training, development and test data for the mod- els we created were supplied by the organizers of SemEval-2023 Task 12 (Muhammad et al., 2023a). The dataset is a collection of tweets labelled with three sentiment classes (negative, positive, and neu- tral) from 14 African languages including Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Mo- roccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorùbá. The Twi language is the focus of this paper. The sentiment of each tweet was determined by a majority vote among three annotators. The tweets are from the the AfriSenti Corpus (Muhammad et al., 2023a) and were curated us- ing the Twitter Academic API, a publicly available Twitter API supporting the retrieval of tweets in 70 languages. However, Amharic is the only African language included in that group of 70. The lack of support for any of the remaining …

Content truncated to 500 words