Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2608–2618 August 1–6, 2021. ©2021 Association for Computational Linguistics 2608 Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities Jinming Zhao∗ School of Information Renmin University of China zhaojinming@ruc.edu.cn Ruichen Li∗ School of Information Renmin University of China ruichen@ruc.edu.cn Qin Jin † School of Information Renmin University of China qjin@ruc.edu.cn Abstract Multimodal fusion has been proved to improve emotion recognition performance in previous works. However, in real-world applications, we often encounter the problem of missing modality, and which modalities will be miss- ing is uncertain. It makes the fixed multimodal fusion fail in such cases. In this work, we pro- pose a unified model, Missing Modality Imag- ination Network (MMIN), to deal with the uncertain missing modality problem. MMIN learns robust joint multimodal representations, which can predict the representation of any missing modality given available modalities under different missing modality conditions. Comprehensive experiments on two bench- mark datasets demonstrate that the unified MMIN model significantly improves emotion recognition performance under both uncertain missing-modality testing conditions and full- modality ideal testing condition. The code will be available at https://github.com/AIM3- RUC/MMIN. 1 Introduction Automatic multimodal emotion recognition is very important to natural human-computer interactions (Fragopanagos and Taylor, 2002). It aims to un- derstand and interpret human emotions expressed through multiple modalities such as speech con- tent, voice tones and facial expression. Previous works have shown that these different modalities are complimentary for emotion expression, and pro- posed many effective multimodal fusion methods to improve the emotion recognition performance (Baltruˇsaitis et al., 2018; Tsai et al., 2019; Zhao et al., 2018). However, in real applications, many common causes can lead to the missing modality problem. For example, the camera is turned off or ∗Equal Contribution †Corresponding Author ... I can’t believe it! Acoustic Visual Textual Missing Emotion Recognition Figure 1: Illustration of a missing modality scenario for multimodal emotion recognition systems. As shown in this video segment, we encounter the missing vi- sual modality problem due to the person’s face was ob- scured by her hands. blocked due to privacy issues; the speech content is unavailable due to automatic speech recognition errors; the voice and text are missing due to the silence of the user; or the faces cannot be detected due to lighting or occlusion issues as shown in Fig- ure 1. Existing multimodal fusion models trained on full-modality samples usually fail when partial modalities are missing (Aguilar et al., 2019; Pham et al., 2019; Cai et al., 2018; Parthasarathy and Sundaram, 2020). The missing modality problem has attracted more research attention in the past years, and the existing solutions for this problem are mainly based on learning joint multimodal representation so that all modality information can be encoded. Han et al. (Han et al., 2019) propose a joint training approach that implicitly fuses multimodal information from auxiliary modalities, which improves the mono- modal emotion recognition performance. …

Content truncated to 500 words