Figure Eight at SemEval-2019 Task 3: Ensemble of Transfer Learning Methods for Contextual Emotion Detection

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 220–224 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 220 Figure Eight at SemEval-2019 Task 3: Ensemble of Transfer Learning Methods for Contextual Emotion Detection Joan Xiao Figure Eight Inc. San Francisco, CA 94103 joan.xiao@gmail.com Abstract This paper describes our transfer learning- based approach to contextual emotion detec- tion as part of SemEval-2019 Task 3. We experiment with transfer learning using pre- trained language models (ULMFiT, OpenAI GPT, and BERT) and fine-tune them on this task. We also train a deep learning model from scratch using pre-trained word embed- dings and BiLSTM architecture with attention mechanism. The ensembled model achieves competitive result, ranking ninth out of 165 teams. The result reveals that ULMFiT per- forms best due to its superior fine-tuning tech- niques. We propose improvements for future work. 1 Introduction Traditionally sentiment analysis attempts to clas- sify the polarity of a given text at the document, sentence, or feature/aspect level, i.e., whether the expressed opinion in the text is positive, negative, or neutral. More advanced sentiment classification looks at emotional states such as “Angry”, “Sad”, and “Happy”. Due to the increasing popularity of social me- dia, over the past years sentiment analysis tasks in SemEval competitions have been mostly fo- cused on twitter (Rosenthal et al., 2014) (Rosen- thal et al., 2015; Nakov et al., 2016; Rosenthal et al., 2017). SemEval-2018 Task 1: Affect in Tweets (Mohammad et al., 2018) includes an ar- ray of subtasks on inferring the emotions (such as joy, fear, valence, and arousal) of a person from his/her tweet. As we increasingly communicate using text messaging applications and digital agents, contex- tual emotion detection in text is gaining impor- tance to provide emotionally aware responses to users. SemEval-2019 Task 3 (Chatterjee et al., 2019) introduces a task to detect contextual emo- tion in conversational text. Deep-learning based approaches have recently dominated the state-of-the-art in sentiment anal- ysis. However, a good performing model often requires large amounts of labeled data and takes many days to train. In computer vision, trans- fer learning has enabled deep learning practition- ers to leverage models that have been pre-trained on ImageNet, MS-COCO, and other large datasets (Razavian et al., 2014; Shelhamer et al., 2017; He et al., 2016; Huang et al., 2017). Fine-tuning such pre-trained models in computer vision has been a far more common practice than training from scratch. In Natural Language Processing (NLP), the most common and simple transfer learning tech- nique is fine-tuning pre-trained word embeddings (Mikolov et al., 2013). These embeddings are used as the first layer of the model on the new dataset, and still require training from scratch with large amounts of labeled data to obtain good perfor- mance. In 2018 several pre-trained language models (ULMFiT, OpenAI GPT and BERT) emerged. These models are trained on very large corpus, and enable robust transfer learning for fine-tuning NLP tasks with little labeled data. In SemEval-2019 Task 3, we apply transfer learning …

Content truncated to 500 words