Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1834–1845, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1834 Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos Nayu Liu1,2, Xian Sun1,2,∗, Hongfeng Yu1, Wenkai Zhang1, Guangluan Xu1 1Key Laboratory of Network Information System Technology, Aerospace Information Research Institute, Chinese Academy of Sciences 2School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences liunayu18@mails.ucas.ac.cn,sunxian@mail.ie.ac.cn Abstract Multimodal summarization for open-domain videos is an emerging task, aiming to gen- erate a summary from multisource informa- tion (video, audio, transcript). Despite the success of recent multiencoder-decoder frame- works on this task, existing methods lack fine- grained multimodality interactions of multi- source inputs. Besides, unlike other multi- modal tasks, this task has longer multimodal sequences with more redundancy and noise. To address these two issues, we propose a multistage fusion network with the fusion for- get gate module, which builds upon this ap- proach by modeling fine-grained interactions between the multisource modalities through a multistep fusion schema and controlling the flow of redundant information between mul- timodal long sequences via a forgetting mod- ule. Experimental results on the How2 dataset show that our proposed model achieves a new state-of-the-art performance. Comprehensive analysis empirically verifies the effectiveness of our fusion schema and forgetting module on multiple encoder-decoder architectures. Spe- cially, when using high noise ASR transcripts (WER>30%), our model still achieves per- formance close to the ground-truth transcript model, which reduces manual annotation cost. 1 Introduction With the popularity of video platforms, personal videos abound on the Internet. Multimodal sum- marization for open-domain videos, first organized as a track of the How2 Challenge at the ICML 2019 workshop, aims to integrate multisource in- formation of videos (video, audio, transcript) into a fluent textual summary. An example can be seen in Figure 1. This study, which uses compressed text description to reflect the salient parts of videos, ∗Corresponding author. is of considerable significance for helping users better retrieve and recommend videos. Existing approaches have obtained promising results. For example, Libovick`y et al. (2018) and Palaskar et al. (2019) utilize multiple encoders to encode videos and audio transcripts and a joint de- coder to decode the multisource encodings, which acquire better performance than single modality structures. Despite the effectiveness of these ap- proaches, they only perform multimodal fusion during the decoding stage to generate a target se- quence, lacking fine-grained interactions between multisource inputs to complete the missing infor- mation of each modality. For example, as shown in Figure 1, text context representations contain- ing birds should be associated with visual semantic information containing parrots to build thorough multimodal representations. Besides, unlike other multimodal tasks such as visual question answering (Antol et al., 2015; Gao et al., 2015) and multimodal machine translation (Elliott et al., 2015; Specia et al., 2016), a ma- jor challenge is that this task has longer input se- quences with more noise and redundancy. The flow of noise information during multimodal fu- sion, such …

Content truncated to 500 words