SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 39–48 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 39 SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi and Puneet Agrawal Microsoft, India {anchatte, kedharn, mejoshi, punagr}@microsoft.com Abstract In this paper, we present the SemEval-2019 Task 3 - EmoContext: Contextual Emotion Detection in Text. Lack of facial expressions and voice modulations make detecting emo- tions in text a challenging problem. For in- stance, as humans, on reading “Why don’t you ever text me!” we can either interpret it as a sad or angry emotion and the same ambigu- ity exists for machines. However, the context of dialogue can prove helpful in detection of the emotion. In this task, given a textual dia- logue i.e. an utterance along with two previ- ous turns of context, the goal was to infer the underlying emotion of the utterance by choos- ing from four emotion classes - Happy, Sad, Angry and Others. To facilitate the participa- tion in this task, textual dialogues from user interaction with a conversational agent were taken and annotated for emotion classes af- ter several data processing steps. A training data set of 30160 dialogues, and two evalu- ation data sets, Test1 and Test2, containing 2755 and 5509 dialogues respectively were released to the participants. A total of 311 teams made submissions to this task. The final leader-board was evaluated on Test2 data set, and the highest ranked submission achieved 79.59 micro-averaged F1 score. Our analysis of systems submitted to the task indicate that Bi-directional LSTM was the most common choice of neural architecture used, and most of the systems had the best performance for the Sad emotion class, and the worst for the Happy emotion class. 1 Introduction Emotions are basic human traits and have been studied by researchers in the fields of psychol- ogy, sociology, medicine, computer science etc. for several years. Some of the prominent work in understanding and categorizing emotions include Ekman’s six class categorization (Ekman, 1992) and Plutchik’s “Wheel of Emotion” (Plutchik and Kellerman, 1986) which suggested eight primary bipolar emotions . In recent times, several Arti- ficial Intelligence (AI) agents like Siri, Cortana, Alexa have emerged and they primarily focus on providing users with assistance on specific tasks such as booking tickets or scheduling meetings etc. However, we believe that for machines and humans to develop a deeper partnership, an In- telligence Quotient (IQ) is not enough. These agents need to also possess an Emotional Quotient (EQ). Social conversational agents like Mitsuku1 or Ruuh 2 (Damani et al., 2018) are experimental agents designed to have human-like persona, and possess a deeper sense of EQ; understanding and expressing emotions is an inherent aspect of these agents. Detecting emotions in textual dialogues is a chal- lenging problem in absence of facial expressions and voice modulations. Moreover, we observed that context of ongoing dialogue can completely change the emotion for an utterance as compared to …

Content truncated to 500 words