Multi-modal Multi-label Emotion Detection with Modality and Label Dependence

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3584–3593, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3584 Multi-modal Multi-label Emotion Detection with Modality and Label Dependence Dong Zhang, Xincheng Ju, Junhui Li, Shoushan Li∗, Qiaoming Zhu, Guodong Zhou School of Computer Science and Technology, Soochow University, China dzhang@suda.edu.cn, xcju@stu.suda.edu.cn {jhli, lishoushan, qmzhu, gdzhou}@suda.edu.cn Abstract As an important research issue in the natural language processing community, multi-label emotion detection has been drawing more and more attention in the last few years. How- ever, almost all existing studies focus on one modality (e.g., textual modality). In this pa- per, we focus on multi-label emotion detec- tion in a multi-modal scenario. In this sce- nario, we need to consider both the depen- dence among different labels (label depen- dence) and the dependence between each pre- dicting label and different modalities (modal- ity dependence). Particularly, we propose a multi-modal sequence-to-set approach to ef- fectively model both kinds of dependence in multi-modal multi-label emotion detection. The detailed evaluation demonstrates the effec- tiveness of our approach. 1 Introduction Emotion detection is to predict emotion categories, such as angry, happy, and surprise, expressed by an utterance of a speaker and has largely encompassed a variety of applications, such as online chatting (Galik and Rank, 2012; Zhang et al., c), news anal- ysis (Li et al., 2015; Zhu et al., 2019) and dialogue systems (Ghosal et al., 2019; Zhang et al., d). Over the last few years, there has been a substantial body of research on emotion detection (Abdul-Mageed and Ungar, 2017; Zhou et al., 2019; Zhang et al., a), where a considerable amount of work has focused on multi-label emotion detection (Li et al., 2015; Yu et al., 2018; Ying et al., 2019). Basically, emotion detection is a multi-label clas- sification problem since one utterance naturally tends to involve more than one emotion category. However, classifying instances with multiple pos- sible categories is sometimes much more difficult than classifying instances with a single label. One ∗Corresponding author �� the hardest thing that we face is �� Textual Modality Visual Modality Acoustic Modality Sad, Disgust Emotions Figure 1: An example of multi-modal instance with multi-label emotion categories in a video segment. main challenge is how to model the label depen- dence in the classification approach. For example, in the utterance as shown in Figure 1, both the Sad and Disgust emotions are more likely to ex- ist, rather than the conflicting emotions of Sad and Happy. Recent studies, such as (Yang et al., 2019) and (Xiao et al., 2019), have begun to address this challenge. However, almost all existing studies in multi- label emotion detection focus on one modality (e.g., textual modality). Only very recently, the research community has become increasingly aware of the need on multi-modal emotion detection (Zadeh et al., 2018b) due to its wide potential applications, e.g., with the massively growing importance of an- alyzing conversations in speech (Gu et al., 2019) and video (Majumder et al., 2019). In …

Content truncated to 500 words