Real-Time Speech Emotion and Sentiment Recognition for Interactive Dialogue Systems

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1042–1047, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Real-Time Speech Emotion and Sentiment Recognition for Interactive Dialogue Systems Dario Bertero, Farhad Bin Siddique, Chien-Sheng Wu, Yan Wan, Ricky Ho Yin Chan and Pascale Fung Human Language Technology Center Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong [dbertero, fsiddique]@connect.ust.hk, b01901045@ntu.edu.tw, ywanad@connect.ust.hk, eehychan@ust.hk, pascale@ece.ust.hk Abstract In this paper, we describe our approach of en- abling an interactive dialogue system to rec- ognize user emotion and sentiment in real- time. These modules allow otherwise con- ventional dialogue systems to have “empathy” and answer to the user while being aware of their emotion and intent. Emotion recognition from speech previously consists of feature en- gineering and machine learning where the first stage causes delay in decoding time. We de- scribe a CNN model to extract emotion from raw speech input without feature engineer- ing. This approach even achieves an impres- sive average of 65.7% accuracy on six emotion categories, a 4.5% improvement when com- pared to the conventional feature based SVM classification. A separate, CNN-based senti- ment analysis module recognizes sentiments from speech recognition results, with 82.5 F- measure on human-machine dialogues when trained with out-of-domain data. 1 Introduction Interactive dialogue systems and chatbots have been around for a while. Some, though not all, systems have statistical and machine learning modules to en- able them to improve overtime. With the pervasive- ness of such systems on mobile devices, expecta- tions of user experience have also increased. We expect human-machine dialogues to get closer to human-human dialogues. One important factor is that we expect machines to understand our emotions and intent and respond with empathy. We propose a module of emotion and sentiment recognition for an interactive dialogue system. This module enables the system to assess the user’s current emotional state and sentiment, and thereby decide the appropriate response at every dialogue state. The dialogue management system handles the mixed-initiative dialogues while taking into account user emotion and sentiment, in addition to query content. Emotion and sentiment recognition enables our system to handle user queries previously unseen in training data. Positive user queries containing positive emotion and sentiment label would have a positive response, and similarly a negatively labeled statement would have a negative response. Examples are shown below: User: I lost my job. Response: Sorry to hear that. Success is in never giving up. User: I just graduated from college! Response: Congratulations! I am happy for you. User: I went on a vacation last month and it was pretty bad, I lost all my luggage Response: That doesn’t sound so good. Hope your next vacation will be a good one. User: My last vacation was amazing, I loved it! Response: That sounds great. I would like to travel with you. Meanwhile, dialogue systems like this need to have real-time recognition of user emotion and sen- timent. Previous …

Content truncated to 500 words