VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9360–9369, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9360 VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles Mingzhe Li1,2,∗, Xiuying Chen1,2,∗, Shen Gao2, Zhangming Chan1,2, Dongyan Zhao1,2 and Rui Yan1,2,† 1 Center for Data Science, AAIS, Peking University,Beijing,China 2 Wangxuan Institute of Computer Technology, Peking University,Beijing,China {li mingzhe,xy-chen,shengao,zhaody,ruiyan}@pku.edu.cn Abstract A popular multimedia news format nowadays is providing users with a lively video and a corresponding news article, which is em- ployed by influential news media including CNN, BBC, and social media including Twit- ter and Weibo. In such a case, automatically choosing a proper cover frame of the video and generating an appropriate textual summary of the article can help editors save time, and read- ers make the decision more effectively. Hence, in this paper, we propose the task of Video- based Multimodal Summarization with Mul- timodal Output (VMSMO) to tackle such a problem. The main challenge in this task is to jointly model the temporal dependency of video with semantic meaning of article. To this end, we propose a Dual-Interaction-based Multimodal Summarizer (DIMS), consisting of a dual interaction module and multimodal generator. In the dual interaction module, we propose a conditional self-attention mecha- nism that captures local semantic information within video and a global-attention mechanism that handles the semantic relationship between news text and video from a high level. Ex- tensive experiments conducted on a large-scale real-world VMSMO dataset1 show that DIMS achieves the state-of-the-art performance in terms of both automatic metrics and human evaluations. 1 Introduction Existing experiments (Li et al., 2017) have proven that multimodal news can significantly improve users’ sense of satisfaction for informativeness. As one of these multimedia data forms, introducing news events with video and textual descriptions is ∗Equal contribution. Ordering is decided by a coin flip. † Corresponding author. 1https://github.com/yingtaomj/VMSMO Figure 1: An example of video-based multimodal sum- marization with multimodal output. becoming increasingly popular, and has been em- ployed as the main form of news reporting by news media including BBC, Weibo, CNN, and Daily Mail. An illustration is shown in Figure 1, where the news contains a video with a cover picture and a full news article with a short textual summary. In such a case, automatically generating multimodal summaries, i.e., choosing a proper cover frame of the video and generating an appropriate textual summary of the article can help editors save time and readers make decisions more effectively. There are several works focusing on multimodal summarization. The most related work to ours is (Zhu et al., 2018), where they propose the task of generating textual summary and picking the most representative picture from 6 input candidates. However, in real-world applications, the input is usually a video consisting of hundreds of frames. Consequently, the temporal dependency in a video cannot be simply modeled by static encoding meth- ods. Hence, in this work, we propose a novel task, Video-based Multimodal Summarization with Mul- timodal Output (VMSMO), …

Content truncated to 500 words