SymantoResearch at SemEval-2019 Task 3: Combined Neural Models for Emotion Classification in Human-Chatbot Conversations

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 330–334 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 330 SymantoResearch at SemEval-2019 Task 3: Combined Neural Models for Emotion Classification in Human-Chatbot Conversations Angelo Basile∗, Marc Franco-Salvador∗, Neha Pawar∗, Sanja ˇStajner∗, Mara Chinea Rios, and Yassine Benajiba Symanto Research, N¨urnberg, Germany {angelo.basile, marc.franco, neha.pawar, sanja.stajner, mara.chinea, yassine.benajiba}@symanto.net Abstract In this paper, we present our participation to the EmoContext shared task on detecting emo- tions in English textual conversations between a human and a chatbot. We propose four neu- ral systems and combine them to further im- prove the results. We show that our neural ensemble systems can successfully distinguish three emotions (SAD, HAPPY, and ANGRY), and separate them from the rest (OTHERS) in a highly-imbalanced scenario. Our best sys- tem achieved a 0.77 F1-score and was ranked fourth out of 165 submissions. 1 Introduction Detecting emotions in text is a key task in many scenarios, such as social listening, personalised marketing, customer caring, or in building emo- tionally intelligent chat-bots: in this last case, the task complexity increases, since a bot’s response might influence the user’s emotion. The EmoContext shared task (Chatterjee et al., 2019) was posed as a sequence classification task. Given a set of three conversational turns (human– bot–human), the goal is to predict the emotion of the third turn. The label space contains the emotions SAD, ANGRY and HAPPY, and the la- bel OTHERS denoting anything else (emotional or non-emotional), as illustrated in Table 1. In this paper, we present our approaches to EmoContext shared task, and describe our best system in details. Additionally, we show that: (a) this task is very difficult even for humans (Sec- tion 2.2); (b) for this task, neural approaches out- perform a strong non-neural baseline (Section 4); (c) an ensemble of neural systems with differ- ∗* The first four authors have contributed equally to this work and are ordered alphabetically. ent architectures significantly outperforms the best neural model in isolation (Section 4). 2 Data The data released by the organisers consist of En- glish user-chatbot interactions occurring in an In- dian chat room. An overview of the dataset is provided in Table 2. It can be seen that the la- bel distribution is highly imbalanced, and different for the training set than for the development (dev) and test sets (a 14:18:18:50 distribution for the training set, and a 5:5:5:85 distribution for the dev and test sets). To overcome this issue we tested three strategies: (1) down-sampling the dataset to its smallest class; (2) up-sampling the emotion- related labels with an in-house dataset; and (3) up-sampling by duplicating a random portion of the dataset. None of these solutions worked, and therefore, we trained our best models using the data provided by the organisers. 2.1 Preprocessing The language of this corpus presents many of the features of micro-blogging language: large use of contractions (e.g. I’m gonna bother), elongations (e.g. a vacation tooooooo!), non-standard use of punctuation (e.g. gonna …

Content truncated to 500 words