Effects of Naturalistic Variation in Goal-Oriented Dialog

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 377–387 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 377 Dual Low-Rank Multimodal Fusion Tao Jin1∗, Siyu Huang2∗, Yingming Li1†, and Zhongfei Zhang3 1College of Information Science & Electronic Engineering, Zhejiang University, China 2Baidu Research, China 3Department of Computer Science, Binghamton University, USA {jint zju,yingming}@zju.edu.cn, huangsiyu@baidu.com, zzhang@binghamton.edu Abstract Tensor-based fusion methods have been proven effective in multimodal fusion tasks. However, existing tensor-based methods make a poor use of the fine-grained temporal dynam- ics of multimodal sequential features. Moti- vated by this observation, this paper proposes a novel multimodal fusion method called Fine- Grained Temporal Low-Rank Multimodal Fu- sion (FT-LMF). FT-LMF correlates the fea- tures of individual time steps between mul- tiple modalities, while it involves multiplica- tions of high-order tensors in its calculation. This paper further proposes Dual Low-Rank Multimodal Fusion (Dual-LMF) to reduce the computational complexity of FT-LMF through low-rank tensor approximation along dual di- mensions of input features. Dual-LMF is con- ceptually simple and practically effective and efficient. Empirical studies on benchmark mul- timodal analysis tasks show that our proposed methods outperform the state-of-the-art tensor- based fusion methods with a similar computa- tional complexity. 1 Introduction Multimodal fusion aims to integrate information of multiple modalities as a compact but informative representation. Multimodal fusion is fundamen- tally significant for real-world multimodal applica- tions like speech translation (Yuhas et al., 1989), emotion recognition (De Silva et al., 1997; Chen et al., 2018), and sentiment analysis (Morency et al., 2011). It is very challenging that it requires cor- relating the semantics of multiple modalities in an effective and efficient way. Recently, several methods have been proposed to learn joint embed- dings of multiple modalities (Fukui et al., 2016; Nojavanasghari et al., 2016; Zadeh et al., 2017). ∗ means equal contribution † corresponding author There are two lines of fusion methods: early fusion and late fusion. In this paper, we mainly focus on the former, which aims to integrate information of different modalities before it is processed by the model. Earlier work on early fusion employs a simple concatenation of input features (P´erez-Rosas et al., 2013; Park et al., 2014; Zadeh et al., 2016b). To construct a more compact representation, (Zadeh et al., 2017) introduces Tensor Fusion Network (TFN) which averages the features of each modal- ity along the temporal dimension and transforms the multimodal features into a high-order tensor which is used for subsequent tasks. Although TFN achieves a better performance than the concate- nation manner, its computational complexity in- creases exponentially with the number of modali- ties. (Liu et al., 2018) further proposes Low-Rank Multimodal Fusion (LMF) which employs low- rank approximation to reconstruct the high-order tensor. However, these tensor-based methods ne- glect the fine-grained temporal dynamics which include rich structured information for multimodal modeling. For example, if the facial expression of a man is happy at time step t, he will speak in a positive tone at time step t + ∆t more likely. The features of …

Content truncated to 500 words