MSR India at SemEval-2020 Task 9: Multilingual Models Can Do Code-Mixing Too

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 951–956 Barcelona, Spain (Online), December 12, 2020. 951 MSR India at SemEval-2020 Task 9: Multilingual Models can do Code-Mixing too Anirudh Srinivasan Microsoft Research, India t-ansrin@microsoft.com∗ Abstract In this paper, we present our system for the SemEval 2020 task on code-mixed sentiment analysis. Our system makes use of large transformer based multilingual embeddings like mBERT. Recent work has shown that these models posses the ability to solve code-mixed tasks in addition to their originally demonstrated cross-lingual abilities. We evaluate the stock versions of these models for the sentiment analysis task and also show that their performance can be improved by using unlabelled code-mixed data. Our submission (username Genius1237) achieved the second rank on the English-Hindi subtask with an F1 score of 0.726. 1 Introduction The task of identifying sentiment from text is extremely important in this age where large volumes of text content are being consumed via social media. The task becomes even more interesting when it comes to bilingual communities as these communities exhibit the phenomenon of code-mixing online (Rijhwani et al., 2017). Existing approaches to tackling this problem have mainly been based on statistical methods (Vilares et al., 2016; Patra et al., 2018). These methods have used features like n-gram counts and TF-IDF vectors along with a linear classifier. There have been very few approaches to this problem using deep learning as the amount of labelled code-mixed data available has always been very less. Methods like the one in Pratapa et al. (2018b) train word embeddings using unlabelled code-mixed data, the availability of which is not as problematic as labelled data, and use these embeddings along with a recurrent neural network based model. Recent advancements in natural language processing have shown that large transformer based models like BERT (Devlin et al., 2019), when pre-trained on large corpora, are easily adaptable for downstream tasks with small datasets. These models even perform well in a cross-lingual manner (Conneau et al., 2018) when pre-trained on corpora spanning multiple languages. Our experiments show that these multilingual models perform well even on code-mixing tasks, having had no exposure to any code-mixing during pre-training. We use such a system to solve the code-mixed sentiment analysis problem. We also show that it’s performance can be improved by using a combination of generated and real code-mixed text. The rest of the paper is organized as follows. Section 2 tasks about the dataset for the task and the pre-processing done to it. Section 3 talks about the different systems we evaluated, with Section 3.3 in particular going into how we improved the multilingual models using code-mixed data. Section 4 describes the performance of the different models and Section 5 concludes our discussion. 2 Dataset and Preprocessing The details about the datasets (Patwa et al., 2020) for both the English-Hindi (En-Hi) and English-Spanish (En-Es) tasks are described in Table 1. The datasets comprise entirely of tweets. The English-Hindi ∗Author can be contacted at anirudhsriniv@gmail.com This work is licensed …

Content truncated to 500 words