Multimodal End-to-End Sparse Model for Emotion Recognition

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5305–5316 June 6–11, 2021. ©2021 Association for Computational Linguistics 5305 Multimodal End-to-End Sparse Model for Emotion Recognition Wenliang Dai*, Samuel Cahyawijaya*, Zihan Liu, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {wdaiai,scahyawijaya,zliucr}@connect.ust.hk, pascale@ece.ust.hk Abstract Existing works on multimodal affective com- puting tasks, such as emotion recognition, gen- erally adopt a two-phase pipeline, first ex- tracting feature representations for each sin- gle modality with hand-crafted algorithms and then performing end-to-end learning with the extracted features. However, the extracted fea- tures are fixed and cannot be further fine-tuned on different target tasks, and manually finding feature extraction algorithms does not general- ize or scale well to different tasks, which can lead to sub-optimal performance. In this pa- per, we develop a fully end-to-end model that connects the two phases and optimizes them jointly. In addition, we restructure the current datasets to enable the fully end-to-end training. Furthermore, to reduce the computational over- head brought by the end-to-end model, we in- troduce a sparse cross-modal attention mecha- nism for the feature extraction. Experimental results show that our fully end-to-end model significantly surpasses the current state-of-the- art models based on the two-phase pipeline. Moreover, by adding the sparse cross-modal attention, our model can maintain performance with around half the computation in the feature extraction part. 1 Introduction Humans show their characteristics through not only the words they use, but also the way they speak and their facial expressions. Therefore, in multimodal affective computing tasks, such as emotion recog- nition, there are usually three modalities: textual, acoustic, and visual. One of the main challenges in these tasks is how to model the interactions be- tween different modalities, as they contain both sup- plementary and complementary information (Bal- trušaitis et al., 2018). * Equal contribution. Code is available at: https://github.com/ wenliangdai/Multimodal-End2end-Sparse Sparse�End2End Hand�Crafted Fully�End2End MFCC Glottal Source Prosody ... Figure 1: An illustration of feature extraction from hand-crafted model (left), fully end-to-end model (mid- dle), and sparse end-to-end model (right). The red dots represent the keypoints extracted by hand-crafted mod- els. The areas formed by red lines represent the regions of interest that are processed by (sparse) end-to-end models to extract the features. In the existing works, we discover that a two- phase pipeline is generally used (Zadeh et al., 2018a,b; Tsai et al., 2018, 2019; Rahman et al., 2020). In the first phase, given raw input data, fea- ture representations are extracted with hand-crafted algorithms for each modality separately, while in the second phase, end-to-end multimodal learning is performed using extracted features. However, there are three major defects of this two-phase pipeline: 1) the features are fixed after extraction and cannot be further fine-tuned on target tasks; 2) manually searching for appropriate feature extrac- tion algorithms is needed for different target tasks; and 3) the hand-crafted …

Content truncated to 500 words