Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 481–492 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 481 Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing Sijie Mai School of Electronics and Information Technology, Sun Yat-sen University {maisj,xingslong} @mail2.sysu.edu.cn Haifeng Hu School of Electronics and Information Technology, Sun Yat-sen University Songlong Xing School of Electronics and Information Technology, Sun Yat-sen University huhaif@mail.sysu.edu.cn Abstract We propose a general strategy named ‘divide, conquer and combine’ for multimodal fusion. Instead of directly fusing features at holis- tic level, we conduct fusion hierarchically so that both local and global interactions are con- sidered for a comprehensive interpretation of multimodal embeddings. In the ‘divide’ and ‘conquer’ stages, we conduct local fusion by exploring the interaction of a portion of the aligned feature vectors across various modal- ities lying within a sliding window, which en- sures that each part of multimodal embeddings are explored sufficiently. On its basis, global fusion is conducted in the ‘combine’ stage to explore the interconnection across local inter- actions, via an Attentive Bi-directional Skip- connected LSTM that directly connects distant local interactions and integrates two levels of attention mechanism. In this way, local inter- actions can exchange information sufficiently and thus obtain an overall view of multimodal information. Our method achieves state-of- the-art performance on multimodal affective computing with higher efficiency. 1 Introduction Multimodal machine learning, as prior research shows (Baltruˇsaitis et al., 2019), always yields higher performance in multimodal tasks compared to the situation where only one modality is in- volved. In this paper, we aim at the multimodal machine learning problem, with an emphasis on multimodal affective computing where the task is to infer human’s opinion from given language, vi- sual and acoustic modalities (Poria et al., 2017a). Finding a feasible and effective solution to learning inter-modality dynamics has been an in- triguing and important problem in multimodal learning (Baltruˇsaitis et al., 2019), where inter- modality dynamics represent complementary in- formation contained in more than one involved Figure 1: Schematic Diagram of our fusion strategy. Here the window size and stride are both set to 2. modality to be detected and analyzed for a more accurate comprehension. For this purpose, a large body of prior work mostly treats the feature vec- tors of the modalities as the smallest units and fuse them at holistic level (Barezi et al., 2018; Poria et al., 2016a, 2017b; Liu et al., 2018). Typical- ly, Zadeh et al. (2017) propose a tensor-based fu- sion method which fuses feature vectors of three modalities using Cartesian product. Despite the effectiveness this type of methods have achieved, they give little consideration to acknowledging the variations across different portions of a feature vector which may contain disparate aspects of in- formation and thus fail to render the fusion proce- dure more specialized. Additionally, they conduct fusion within one step, which can be intractable in some scenarios where the fusion …

Content truncated to 500 words