Palomino-Ochoa at SemEval-2020 Task 9: Robust System Based on Transformer for Code-Mixed Sentiment Classification

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 963–967 Barcelona, Spain (Online), December 12, 2020. 963 Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification Daniel Palomino Dept. of Computer Science Universidad Cat´olica San Pablo Arequipa, Peru daniel.palomino.paucar@ucsp.edu.pe Jos´e Ochoa-Luna Dept. of Computer Science Universidad Cat´olica San Pablo Arequipa, Peru jeochoa@ucsp.edu.pe Abstract We present a transfer learning system to perform a mixed Spanish-English sentiment classification task. Our proposal uses the state-of-the-art language model BERT and embed it within a ULMFiT transfer learning pipeline. This combination allows us to predict the polarity detection of code- mixed (English-Spanish) tweets. Thus, among 29 submitted systems, our approach (referred to as dplominop) is ranked 4th on the Sentimix Spanglish test set of SemEval 2020 Task 9. In fact, our system yields the weighted-F1 score value of 0.755 which can be easily reproduced — the source code and implementation details are made available. 1 Introduction Sentiment Analysis is one of the most active research areas in Natural Language Processing (NLP), web mining and social media analytics (Tang et al., 2016). In particular, its polarity detection task has been extensively researched since 2002 (Liu, 2012). Polarity detection involves to determine whether a given text, containing an opinion, is positive, negative or neutral. In order to solve this task, several dictionary-based methods were proposed in the past (Liu, 2012). However, Machine Learning (ML) approaches have been the ones with more impact regarding the-state-of-the-art (Liu, 2015). Moreover, traditional ML approaches based on feature engineering and modelling have been consistently improved by Deep Learning methods (Zhang et al., 2018). As a rule, to apply Deep Learning in sentiment analysis input text should be encoded in some way, for instance using word embeddings. Word embeddings are useful because allow us to encode semantic similarities among words. One can obtain a word vector representation by training a large corpus using algorithms such as Word2vec (Mikolov et al., 2013), Glove (Pennington et al., 2014) and FastText (Bojanowski et al., 2017), to name a few. Nowadays, this kind of text representation has evolved to a language model encoding. The idea is to use language context in order to better encode words and characters. Overall, the aim of this encoding is to transfer the knowledge embodied in the language model to address a specific task. Thus, in this paper we focus on transfer learning: we make use of a pre-trained language model and then we applied it in sentiment classification (The aim is to predict the correct sentiment classification of a given code-mixed tweet). In order to do so, our work combines two powerful frameworks: the Universal Language Model Fine-tuning (ULMFiT) (Howard and Ruder, 2018) and BERT (Devlin et al., 2018). ULMFiT is the basis for our transfer learning strategy and it has proven to have an impressive perfor- mance on several English text classification tasks—It also has obtained optimal results for the Spanish language (Palomino and Ochoa-Luna, 2019). On the other hand, BERT is currently the-state-of-the-art …

Content truncated to 500 words