Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1304–1309 Barcelona, Spain (Online), December 12, 2020. 1304 Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis Vinay Gopalan Reed College gopalanvinay@gmail.com Mark Hopkins Reed College hopkinsm@reed.edu Abstract We explore the task of sentiment analysis on Hinglish (code-mixed Hindi-English) tweets as participants of Task 9 of the SemEval-2020 competition, known as the SentiMix task. We had two main approaches: 1) applying transfer learning by fine-tuning pre-trained BERT models and 2) training feedforward neural networks on bag-of-words representations. During the evaluation phase of the competition, we obtained an F-score of 71.3% with our best model, which placed 4th out of 62 entries in the official system rankings. 1 Introduction The Internet today has a vast collection of data in various forms, including text and images. A big part of this data comes from various social media platforms, which enable users to share thoughts about their daily experiences. The millions of tweets, updates, location check-ins, likes, and dislikes that users share every day on different platforms form a large bank of opinionated data. Extracting sentiment from this data, though immensely useful, is also challenging, giving rise to the NLP task known as sentiment analysis. Several models have been proposed to perform this task over the years, such as those built on top of the Recursive Neural Tensor Network (Socher et al., 2013) or the more recent BERT model (Devlin et al., 2018). Although most of these language technologies are built for the English language, a lot of Internet data comes from multilingual speakers who combine English with other languages when they use social media. Thus, it is also important to study sentiment analysis for this so-called ‘code-mixed’ social media text. In this paper, we explore sentiment analysis on code-mixed Hinglish (Hindi-English) tweets, specifically as a participant in Task 9 of SemEval-2020 (Patwa et al., 2020). We had two main strategies. We began by fine-tuning pre-trained BERT models to our target task. Dur- ing this initial phase, we observed that the accuracies yielded by bert-base, bert-multilingual and bert-chinese on the validation data were approximately the same. This made us hypothesize that the pre-trained weights were mostly unhelpful for the task. To test this, we attempted to recreate the BERT fine-tuning results only using feedforward neural networks trained on a bag-of-words (BoW) representation. We found that the the results of fine-tuning bert-large (24 layers) could be approximated by a 2-layer BoW feedforward neural network, as our best-performing fine-tuned model had an accuracy of 63.9% and our best-performing bag-of-words model had an accuracy of 60.0% on the validation corpus. In the evaluation phase of the competition, our fine-tuned model had an F-score of 69.9% without bagging (Breiman, 1996) and 71.3% with bagging. Once the official system rankings were published on April 6, 20201, our best submission (CodaLab username: gopalanvinay) placed 4th out of 62 entries. All code needed to recreate our results can be found here2. This work is licensed under …

Content truncated to 500 words