EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4096–4113 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4096 EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems Shutong Feng, Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Michael Heck, Carel van Niekerk and Milica Gaˇsi´c Heinrich Heine University D¨usseldorf Universit¨atsstraße 1, 40225 D¨usseldorf, Germany {fengs, lubis, geishaus, linh, heckmi, niekerk, gasic}@hhu.de Abstract The ability to recognise emotions lends a conversational artificial intelligence a human touch. While emotions in chit-chat dialogues have received substantial attention, emotions in task-oriented dialogues remain largely unaddressed. This is despite emotions and dialogue success having equally important roles in a natural system. Existing emotion-annotated task-oriented corpora are limited in size, label richness, and public availability, creating a bottleneck for downstream tasks. To lay a foun- dation for studies on emotions in task-oriented dialogues, we introduce EmoWOZ, a large-scale manually emotion-annotated corpus of task-oriented dialogues. EmoWOZ is based on MultiWOZ, a multi-domain task-oriented dialogue dataset. contains more than 11K dialogues with more than 83K emotion annotations of user utterances. In addition to Wizard-of-OzIt dialogues from MultiWOZ, we collect human-machine dialogues within the same set of domains to sufficiently cover the space of various emotions that can happen during the lifetime of a data-driven dialogue system. To the best of our knowledge, this is the first large-scale open-source corpus of its kind. We propose a novel emotion labelling scheme, which is tailored to task-oriented dialogues. We report a set of experimental results to show the usability of this corpus for emotion recognition and state tracking in task-oriented dialogues. Keywords: Emotion Recognition in Conversations, Task-oriented Dialogues 1. Introduction Incorporating human intelligence into conversational artificial intelligence (AI) has been a challenging and long-term goal (Picard, 1997). Emotional intelligence, defined as the ability to regulate, perceive, assimilate, and express emotions, is a key component of general intelligence (Mayer et al., 1999). Such emotion aware- ness can help the conversational AI generate more emo- tionally and semantically appropriate responses (Zhou et al., 2018). Dialogue systems generally fall into two classes. Task- oriented systems converse with users to help complete tasks determined by user goals. Chit-chat systems are set up to mimic the unstructured conversations or ‘chats’ characteristic of human-human interaction (Ju- rafsky and Martin, 2009). Chat-oriented systems are typically modelled in a supervised fashion with large available corpora (Vinyals and Le, 2015). In con- trast, task-oriented systems track the user goal through- out the dialogue and a policy is typically trained via some form of reinforcement learning (RL) to conduct dialogue towards successful goal completion (Young, 2002). Moreover, the scope of the dialogue can also be extended during this process, e.g. by adding new do- mains to the dialogue system (Madotto et al., 2021). Consequently, the distribution of data from which a task-oriented system learns can change. Emotions appear in both chit-chat and task-oriented di- alogues. However, the cause of emotion may differ as well as their role. Chit-chat …

Content truncated to 500 words