Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1092–1102 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video Haoran Li1,2, Junnan Zhu1,2, Cong Ma1,2, Jiajun Zhang1,2 and Chengqing Zong1,2,3 1 National Laboratory of Pattern Recognition, CASIA, Beijing, China 2 University of Chinese Academy of Sciences, Beijing, China 3 CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China {haoran.li, junnan.zhu, cong.ma, jjzhang, cqzong}@nlpr.ia.ac.cn Abstract The rapid increase in multimedia data transmission over the Internet necessitates the multi-modal summarization (MMS) from collections of text, image, audio and video. In this work, we propose an extrac- tive multi-modal summarization method that can automatically generate a textual summary given a set of documents, im- ages, audios and videos related to a specif- ic topic. The key idea is to bridge the se- mantic gaps between multi-modal content. For audio information, we design an ap- proach to selectively use its transcription. For visual information, we learn the joint representations of text and images using a neural network. Finally, all of the multi- modal aspects are considered to generate the textual summary by maximizing the salience, non-redundancy, readability and coverage through the budgeted optimiza- tion of submodular functions. We further introduce an MMS corpus in English and Chinese, which is released to the public1. The experimental results obtained on this dataset demonstrate that our method out- performs other competitive baseline meth- ods. 1 Introduction Multimedia data (including text, image, audio and video) have increased dramatically recently, which makes it difficult for users to obtain important in- formation efficiently. Multi-modal summarization (MMS) can provide users with textual summaries that can help acquire the gist of multimedia data in a short time, without reading documents or watch- ing videos from beginning to end. 1http://www.nlpr.ia.ac.cn/cip/jjzhang.htm The existing applications related to MMS in- clude meeting record summarization (Erol et al., 2003; Gross et al., 2000), sport video sum- marization (Tjondronegoro et al., 2011; Hasan et al., 2013), movie summarization (Evangelopou- los et al., 2013; Mademlis et al., 2016), pictorial storyline summarization (Wang et al., 2012), time- line summarization (Wang et al., 2016b) and social multimedia summarization (Del Fabro et al., 2012; Bian et al., 2013; Schinas et al., 2015; Bian et al., 2015; Shah et al., 2015, 2016). When summariz- ing meeting recordings, sport videos and movies, such videos consist of synchronized voice, visual and captions. For the summarization of pictorial storylines, the input is a set of images with text descriptions. None of these applications focus on summarizing multimedia data that contain asyn- chronous information about general topics. In this paper, as shown in Figure 1, we propose an approach to a generate textual summary from a set of asynchronous documents, images, audios and videos on the same topic. Since multimedia data are heterogeneous and contain more complex information than pure tex- t does, MMS faces a great challenge in address- ing the semantic gap between different modali- ties. …

Content truncated to 500 words