WESSA at SemEval-2020 Task 9: Code-Mixed Sentiment Analysis Using Transformers

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1342–1347 Barcelona, Spain (Online), December 12, 2020. 1342 WESSA at SemEval-2020 Task 9: Code-Mixed Sentiment Analysis using Transformers Ahmed Sultan WideBot a.sultan@widebot.ai Amina Gaber WideBot amina.gaber@widebot.ai Mahmoud Salim WideBot m.salim@widebot.ai Islam El Hosary WideBot islam@widebot.ai Abstract In this paper, we describe our system submitted for SemEval 2020 Task 9, Sentiment Analysis for Code-Mixed Social Media Text alongside other experiments. Our best performing system is a Transfer Learning-based model that fine-tunes XLM-RoBERTa, a transformer-based multilingual masked language model, on monolingual English and Spanish data and Spanish-English code- mixed data. Our system outperforms the official task baseline by achieving a 70.1% average F1-Score on the official leaderboard using the test set. For later submissions, our system manages to achieve a 75.9% average F1-Score on the test set using CodaLab username “ahmed0sultan”. 1 Introduction Microblogging websites have been a huge source of data containing different kinds of information. Since the users on these microblogging websites tend to write informal real-time messages, they also tend to mix languages as they are just being spontaneous and want to ease the communication or they are just multilingual or non-native language speakers who mix between their native language and the language they are trying to use Patwa et al. (2020). This type of writing is called Code-Mixing or Code-Switching and it could be defined as the phenomenon of mixing the vocabulary and syntax of multiple languages in the same sentence Lal et al. (2019). Sentiment Analysis (SA) is the task of detecting, extracting, and classifying sentiment and opinions Montoyo et al. (2012). SA can help in structuring data of public opinions about products, brands or any topic that people can express opinions about, to be used in a very wide set of practical applications varying from political use, e.g. monitoring public events Tumasjan et al. (2010) to commercial use, e.g. making decisions in the stock market Jansen et al. (2009). The task of monolingual sentiment analysis has been a well-studied topic in the literature over the past few decades. However, little attention has been directed to SA based on code-mixed data. In SemEval-2020 Task 9: Sentiment Analysis for Code-Mixed Social Media Text Patwa et al. (2020), the organizers provide a dataset of Code-Mixed tweets with word-level language labels that we will explore in Section 2.1, and with the following sentiment labels: positive, negative, neutral. Given a code-mixed text, the task is to classify the overall sentiment of the input text to one of the three sentiment labels mentioned above. The official evaluation metric for this task is Average F1-Score. We report the experiments made only on Spanish-English (Spanglish) data, whereas SemEval-2020 Task 9 contains Hindi-English (Hinglish) data as well. The challenges of this shared task could be summarized as follows: a) The relatively small dataset provided makes it hard to train complex models b) the target classes distribution is imbalanced in the training data c) the characteristics of social media text pose difficulties such as out-of-vocabulary words …

Content truncated to 500 words