Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 978–990 November 20–23, 2022. ©2022 Association for Computational Linguistics 978 Meta-Learning based Deferred Optimisation for Sentiment and Emotion aware Multi-modal Dialogue Act Classification Tulika Saha∗, Aditya Prakash Patra†, Sriparna Saha†, Pushpak Bhattacharyya‡ ∗University of Liverpool, United Kingdom †Indian Institute of Technology Patna, India ‡Indian Institute of Technology Bombay, India (sahatulika15,sriparna.saha,pushpakbh)@gmail.com Abstract Dialogue Act Classification (DAC) that deter- mines the communicative intention of an ut- terance has been investigated widely over the years as a standalone task. But the emotional state of the speaker has a considerable effect on its pragmatic content. Sentiment as a hu- man behavior is also closely related to emo- tion and one aids in the better understanding of the other. Thus, their role in identification of DAs needs to be explored. As a first step, we extend the newly released multi-modal EMO- TyDA dataset to enclose sentiment tags for each utterance. In order to incorporate these multiple aspects, we propose a Dual Attention Mecha- nism (DAM) based multi-modal, multi-tasking conversational framework. The DAM mod- ule encompasses intra-modal and interactive inter-modal attentions with multiple loss op- timization at various hierarchies to fuse mul- tiple modalities efficiently and learn general- ized features across all the tasks. Addition- ally, to counter the class-imbalance issue in dia- logues, we introduce a 2-step Deferred Optimi- sation Schedule (DOS) that involves Meta-Net (MN) learning and deferred re-weighting where the former helps to learn an explicit weighting function from data automatically and the latter deploys a re-weighted multi-task loss with a smaller learning rate. Empirically, we establish that the joint optimisation of multi-modal DAC, SA and ER tasks along with the incorporation of 2-step DOS and MN learning produces better results compared to its different counterparts and outperforms state-of-the-art model. 1 Introduction Dialogue Act Classification (DAC) constitutes an important means for understanding a speaker’s communicative intention (for example, question, command, apology etc.) in any Dialogue System (Stolcke et al., 2000), (Papalampidi et al., 2017). Thus, DA seeks to analyze the pragmatics of a conversation instead of just its literal meaning. Au- thors of (Saha et al., 2020b) went a step ahead and established in a multi-modal setting (including text, audio and video) that a speaker’s true communica- tive content is greatly influenced by its emotional state of mind (Barrett et al., 1993). Utterances such as “Oh sure" or “Ya why not" can be understood as “agreement" or “disagreement" (if implied sarcasti- cally). However, the emotional state of the speaker might enclose cues giving it another definition alto- gether. Sentiment and emotion are frequently viewed as two different entities (Do et al., 2019; Hossain and Muhammad, 2019; Majumder et al., 2019) etc., but are often interpreted in a similar way and are therefore used interchangeably due to their sub- jective character. But sentiment and emotion are not literally the same, but are strongly linked. …

Content truncated to 500 words