HinglishNLP at SemEval-2020 Task 9: Fine-tuned Language Models for Hinglish Sentiment Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 934–939 Barcelona, Spain (Online), December 12, 2020. 934 HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection Meghana Bhange Verloop.io Bengaluru meghana@verloop.io Nirant Kasliwal Verloop.io Bengaluru hi@nirantk.com Abstract Sentiment analysis for code-mixed social media text continues to be an under-explored area. This work adds two common approaches: fine-tuning large transformer models and sample efficient methods like ULMFiT (Howard and Ruder, 2018). Prior work demonstrates the efficacy of classical ML methods for polarity detection. Fine-tuned general-purpose language representation models, such as those of the BERT family are benchmarked along with classical machine learning and ensemble methods. We show that NB-SVM beats RoBERTa by 6.2% (relative) F1. The best performing model is a majority-vote ensemble which achieves an F1 of 0.707. The leaderboard submission was made under the codalab username nirantk, with F1 of 0.689. 1 Introduction Code-mixing or code-switching refers to the use of two or more languages or speech variants together (Contini-Morava, 1995). This is commonly observed in informal conversations, especially those on social media, e.g. Twitter (Rudra et al., 2016) (Rijhwani et al., 2017). While a small body of work does exist on code-mixing detection, this task focuses on polarity detection (Sentiment Analysis). We demonstrate the impressive performance of transfer learning for the task of sentiment detection in code-mixed context. We discuss limitations of existing deep learning pre-trained models which are trained on ”monolingual” text – which has sentences in only one language. In this work, we demonstrate how it is beneficial to fine-tune the language model (LM) for a code-mixed setting like Hinglish, when Hindi is written in the Roman script. The labelled data for the sentiment classifier is from the task paper — (Patwa et al., 2020). It consists of a total of 17000 tweets. The sentiment labels are positive, negative, or neutral, and the code-mixed languages are English-Hindi. The train data after the split has 14k tweets. The validation and test data contain 3k tweets each. 2 Data We used two primary datasets. The data used for fine-tuning LM is from Twitter stream. The tweet stream data was used which contains ∼1.9 million Hinglish tweets. We manually sampled 3k tweets to verify what fraction of them are Hinglish. This dataset consists of ∼86.5% Hinglish tweets. The 17k tweet Sentimix data (Patwa et al., 2020) was used to further fine-tune the sentiment classifier. The Sentimix data contains 14,000 tagged tweets marked as positive, negative and neutral for training, and 3,000 for development and testing each. 2.1 Mining and Filtering Twitter Corpus Pre-training LM require large datasets. In order to enable this, we gathered ∼1.9 Million tweets from the Twitter 1% sample stream for the entire year of 2018. We curated a seed dictionary of Hinglish words and their spelling variants. Next, we calculate the Jaccard Index (Jaccard, 1912) between our seed dictionary This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 935 and every tweet. For values above 0.6, we mark …

Content truncated to 500 words