Towards Emotion-aided Multi-modal Dialogue Act Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361–4372 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4361 Towards Emotion-aided Multi-modal Dialogue Act Classification Tulika Saha∗, Aditya Prakash Patra∗, Sriparna Saha, Pushpak Bhattacharyya Department of Computer Science and Engineering, Indian Institute of Technology Patna, India (sahatulika15,aditya.prakash.patra1997)@gmail.com (sriparna.saha,pushpakbh)@gmail.com Abstract The task of Dialogue Act Classification (DAC) that purports to capture communicative intent has been studied extensively. But these stud- ies limit themselves to text. Non-verbal fea- tures (change of tone, facial expressions etc.) can provide cues to identify DAs, thus stress- ing the benefit of incorporating multi-modal inputs in the task. Also, the emotional state of the speaker has a substantial effect on the choice of the dialogue act, since conversations are often influenced by emotions. Hence, the effect of emotion too on automatic identifica- tion of DAs needs to be studied. In this work, we address the role of both multi-modality and emotion recognition (ER) in DAC. DAC and ER help each other by way of multi-task learn- ing. One of the major contributions of this work is a new dataset- multimodal Emotion aware Dialogue Act dataset called EMOTyDA, collected from open-sourced dialogue datasets. To demonstrate the utility of EMOTyDA, we build an attention based (self, inter-modal, inter-task) multi-modal, multi-task Deep Neu- ral Network (DNN) for joint learning of DAs and emotions. We show empirically that multi- modality and multi-tasking achieve better per- formance of DAC compared to uni-modal and single task DAC variants. 1 Introduction Dialogue Act Classification (DAC) is concerned with deciding the type i.e., communicative inten- tion (question, statement, command etc.) of the speaker’s utterance. DAC is very important in the context of discourse structure, which in turn sup- ports intelligent dialogue systems, conversational speech transcription and so on. Considerable works have been done on classical Machine Learning (ML) based DAC (Jurafsky et al., 1997), (Stolcke et al., 2000), (Verbree et al., 2006), etc. and Deep ∗The authors have contributed equally. Learning (DL) based DAC (Kalchbrenner and Blun- som, 2013), (Papalampidi et al., 2017), (Liu et al., 2017), (Ribeiro et al., 2019), (Ortega et al., 2019), (Saha et al., 2019) etc. Humans are emotional entities. A speaker’s emo- tional state considerably influences or affects its intended content or its pragmatic content (Barrett et al., 1993). An utterance such as “Okay sure” or “Ya right” (say) can be considered as “agreement” or- in case of sarcasm- “disagreement”. For expres- sive DAs such as “greeting”, “thanking”, “apolo- gizing” etc., the speaker’s feeling or emotion can assist in recognizing true communicative intent and vice-versa. Thus, it is important to consider the speaker’s emotion when deciding on the DA. There is considerable work on ER (Cowie et al., 2001), (Jain et al., 2018), (Zhang et al., 2018), etc. and adapting the Virtual Agents (VAs) to act ac- cordingly (Huang et al., 2018), (Zhou et al., 2018), (Fung et al., 2018), etc. But very little research has been done, that addresses the impact of emotion …

Content truncated to 500 words