CTFN: Hierarchical Learning for Multimodal Sentiment Analysis Using Coupled-Translation Fusion Network

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5301–5311 August 1–6, 2021. ©2021 Association for Computational Linguistics 5301 CTFN: Hierarchical Learning for Multimodal Sentiment Analysis Using Coupled-Translation Fusion Network Jiajia Tang1∗, Kang Li1∗, Xuanyu Jin1, Andrzej Cichocki2 3, Qibin Zhao4, Wanzeng Kong1† 1Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, School of Computer Science and Technology, Hangzhou Dianzi University, China 2Skolkovo Institute of Science and Technology, Moscow, Russia 3Systems Research Institute, Polish Academy of Science, Warsaw, Poland 4Center for Advanced Intelligence Project, RIKEN {hdutangjiajia, Jxuanyu599}@163.com {likang bro, kongwanzeng}@hdu.edu.cn {a.cichocki, qibin.zhao}@riken.jp Abstract Multimodal sentiment analysis is the challeng- ing research area that attends to the fusion of multiple heterogeneous modalities. The main challenge is the occurrence of some missing modalities during the multimodal fusion pro- cedure. However, the existing techniques re- quire all modalities as input, thus are sensi- tive to missing modalities at predicting time. In this work, the coupled-translation fusion network (CTFN) is firstly proposed to model bi-direction interplay via couple learning, en- suring the robustness in respect to missing modalities. Specifically, the cyclic consistency constraint is presented to improve the transla- tion performance, allowing us directly to dis- card decoder and only embraces encoder of Transformer. This could contribute to a much lighter model. Due to the couple learning, CTFN is able to conduct bi-direction cross- modality intercorrelation parallelly. Based on CTFN, a hierarchical architecture is further es- tablished to exploit multiple bi-direction trans- lations, leading to double multimodal fusing embeddings compared with traditional trans- lation methods. Moreover, the convolution block is utilized to further highlight explicit interactions among those translations. For evaluation, CTFN was verified on two mul- timodal benchmarks with extensive ablation studies. The experiments demonstrate that the proposed framework achieves state-of-the-art or often competitive performance. Addition- ally, CTFN still maintains robustness when considering missing modality. 1 Introduction Sentiment analysis has witnessed many significant advances in the artificial intelligence community, in which text (Yadollahi et al., 2017), visual (Kahou et al., 2016), and acoustic (Luo et al., 2019) modal- ities are primarily employed to the related research ∗∗Equal contribution †Corresponding author: Wanzeng Kong respectively, allowing to exploit the human emo- tional characteristic and intention effectively (Deng et al., 2018). Intuitively, due to the consistency and complementarity among different sources, the joint representation attend to reason about multimodal messages, which are capable of boosting the perfor- mance of the specific task (Pan et al., 2016; Gebru et al., 2017; Al Hanai et al., 2018). Multimodal fusion procedure is to incorporate multiple knowledge for predicting a precise and proper outcome (Baltruˇsaitis et al., 2018). Histori- cally, the existing fusion has been done generally by leveraging the model-agnostic process, consid- ering the early fusion, late fusion, and hybrid fu- sion technique (Poria et al., 2017a). Among those, early fusion focussed on the concatenation of the unimodal presentation (D’mello and Kory, 2015). On the contrast, late fusion performs the integra- tion at the …

Content truncated to 500 words