FiSSA at SemEval-2020 Task 9: Fine-tuned for Feelings

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1239–1246 Barcelona, Spain (Online), December 12, 2020. 1239 FiSSA at SemEval-2020 Task 9: Fine-tuned For Feelings Bertelt Braaksma Richard Scholtens Stan van Suijlekom Remy Wang Ahmet ¨Ust¨un University of Groningen Groningen, the Netherlands {b.braaksma.1, j.f.p.scholtens, s.van.suijlekom, r.wang.7}@student.rug.nl, a.ustun@rug.nl Abstract In this paper, we present our approach for sentiment classification on Spanish-English code- mixed social media data in the SemEval-2020 Task 9. We investigate performance of various pre-trained Transformer models by using different fine-tuning strategies. We explore both mono- lingual and multilingual models with the standard fine-tuning method. Additionally, we propose a custom model that we fine-tune in two steps: once with a language modeling objective, and once with a task-specific objective. Although two-step fine-tuning improves sentiment classifi- cation performance over the base model, the large multilingual XLM-RoBERTa model achieves best weighted F1-score with 0.537 on development data and 0.739 on test data. With this score, our team jupitter placed tenth overall in the competition. 1 Introduction Code-mixing is a phenomenon in which two or more languages are used in a single utterance. It occurs at various levels of linguistic structure: across sentences (i.e., inter-sentential), within a sentence (i.e.,intra- sentential), or at the word/morpheme level. In addition to spoken language, this phenomenon has become especially prevalent on social media. As monolingual systems cannot deal with the code-mixed data, it poses a major challenge for even the most standard NLP tasks. To this end, SemEval 2020 Task 9 (Patwa et al., 2020) proposes the sentiment analysis task for code-mixed social media text, specifically on English-Spanish (Spanglish) and English-Hindi (Hinglish) language pairs. In this paper, we present our approach called Fine-tuned Spanglish Sentiment Analysis, or FiSSA for short. We focus on various pre-trained language models for Spanglish sentiment classification by fine-tuning their contextualized word embeddings. By doing so, we examine two challenging aspects of code-mixed language processing: (a) Multilinguality, (b) Domain. We firstly compare monolingual models with their multilingual counterparts to evaluate the multilingual solution on code-switching data, considering the first aspect. Secondly, to see the domain effect, we further fine-tune the multilingual model on domain-specific unlabeled data. Finally, we use the most recent state-of-the-art pre-trained model and compare it to our custom model with domain information. Our research shows that fine-tuning a pre-trained language model is a good choice compared to the standard BLSTM model when training data is limited. However, on code-mixed data, monolingual pre- trained models tend to perform better on different portions of the data depending on the use of languages. As the best alternative, a large multilingual model provides better generalization and results in a stronger performance. 2 Background Code-Mixed Text Processing Only a limited amount of research has been done in the field of senti- ment analysis on Spanish-English social media data. However, some writing has been done regarding Spanish-English code-mixing for other NLP tasks, such as part-of-speech tagging (Solorio and Liu, 2008) and language identification (Solorio et al., 2014). In the first shared task of language …

Content truncated to 500 words