Howard University Computer Science at SemEval-2023 Task 12: A 2-Step System Design for Multilingual Sentiment Classification with Language Identification

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2153–2159 July 13-14, 2023 ©2023 Association for Computational Linguistics Howard University Computer Science at SemEval-2023 Task 12: A 2-Step System Design for Multilingual Sentiment Classification with Language Identification Saurav Aryal and Howard Prioleau EECS, Howard University, Washington, DC 20059, USA https://howard.edu/ saurav.aryal@howard.edu and howard.prioleau@bison.howard.edu Abstract The recent release of the AfriSenti-SemEval shared Task 12 has made available 14 new datasets annotated for sentiment analysis on African Languages. We proposed and evaluated two approaches to this task, Delta TF-IDF, and a proposed Language-Specific Model Fusion System using Language Identification. Both produced comparable or better classification performance than the current state-of-art mod- els on this task: AfriBERTa, AfroXLMR, and AfroLM. 1 Introduction AfriSenti-SemEval Shared Task 12 (Muham- mad et al., 2023b,a) provides an landmark dataset African language sentiment analysis. The task provided 13 datasets comprising 12 different languages, each being a dataset and a dataset composed of all the languages. The 12 African Langauges covered are Hausa(HA), Yoruba(YO), Igbo(IG), Nigerian Pigdin(PCM), Amharic(AM), Algerian Arabic(DZ), Mo- roccan Arabic/Darija(MA), Swahili(SW), Kinyarwanda(KR), Twi(TWI), Mozambican Portuguese(PT), and Xitsonga(Mozambique Dialect) (TS). These dataset languages cover a wide range of different regions of Africa. Each tweet in each dataset is manually annotated between 3 positive, negative, and neutral labels. Some datasets contain code-mixed data (data that contains two or more languages) and transliteration (converting text from one script to another that involves swapping letters). While most of these languages have a limited amount of corpus, to our knowledge, some languages, such as Xitsonga, have labeled sentiment analysis datasets created for the first time (Aryal et al., 2023). This paper aims to contribute to the NLP and sen- timent analysis literature for African languages by going in-depth into our modeling practices on the AfriSenti-SemEval (Muhammad et al., 2023b,a) Shared Task 12 and showcasing our results. We propose a two-step modeling approach by first per- forming language detection to identify the text’s language and following up with a language-specific model for final sentiment classification. We were able to discover the effectiveness of Delta TF-IDf on low-resource African languages. This proves useful since it requires a lot less data to be just competitive or even outperform the much larger comparable transformers models. Further- more, building on this, we were able to propose and develop a high-performing algorithm by in- corporating language detection; which allowed us to leverage the best-performing models of specific languages since they provide the best performance. 2 Relevant Works This section will cover the models that we trained and utilized to build up our proposed system de- sign. 2.1 Models Some researchers suggest that approximately 30% of all current languages are of African origin. De- spite this, few large multilingual models cover African languages, with most models covering fewer than five officially recognized African lan- guages. Therefore, this section will focus on three models aimed at addressing this issue, which are AfriBERTa (Ogueji et al., 2021), AfroXLMR (Al- abi et al., 2022), and AfroLM (Dossou et …

Content truncated to 500 words