Multimodal Abstractive Summarization for How2 Videos

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6587–6596 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6587 Multimodal Abstractive Summarization for How2 Videos Shruti Palaskar1 Jindˇrich Libovický2 Spandana Gella3∗ Florian Metze1 1School of Computer Science, Carnegie Mellon University 2Faculty of Mathematics and Physics, Charles University 3 Amazon AI spalaska@cs.cmu.edu, libovicky@ufal.mff.cuni.cz sgella@amazon.com, fmetze@cs.cmu.edu Abstract In this paper, we study abstractive summariza- tion for open-domain videos. Unlike the tra- ditional text news summarization, the goal is less to “compress” text information but rather to provide a fluent textual summary of infor- mation that has been collected and fused from different source modalities, in our case video and audio transcripts (or text). We show how a multi-source sequence-to-sequence model with hierarchical attention can integrate infor- mation from different modalities into a coher- ent output, compare various models trained with different modalities and present pilot ex- periments on the How2 corpus of instructional videos. We also propose a new evaluation met- ric (Content F1) for abstractive summarization task that measures semantic adequacy rather than fluency of the summaries, which is cov- ered by metrics like ROUGE and BLEU. 1 Introduction In recent years, with the growing popularity of video sharing platforms, there has been a steep rise in the number of user-generated instructional videos shared online. With the abundance of videos online, there has been an increase in demand for efficient ways to search and retrieve relevant videos (Song et al., 2011; Wang et al., 2012; Otani et al., 2016; Torabi et al., 2016). Many cross-modal search applications rely on text associated with the video such as description or title to find relevant content. However, often videos do not have text meta-data associated with them or the existing ones do not provide clear information of the video con- tent and fail to capture subtle differences between related videos (Wang et al., 2012). We address this by aiming to generate a short text summary of the video that describes the most salient content of the ∗*Work done while SG was at University of Edinburgh video. Our work benefits users through better con- textual information and user experience, and video sharing platforms with increased user engagement by retrieving or suggesting relevant videos to users and capturing their attention. Summarization is a task of producing a shorter version of the content in the document while pre- serving its information and has been studied for both textual documents (automatic text summariza- tion) and visual documents such as images and videos (video summarization). Automatic text sum- marization is a widely studied topic in natural lan- guage processing (Luhn, 1958; Kupiec et al., 1995; Mani, 1999); given a text document the task is to generate a textual summary for applications that can assist users to understand large documents. Most of the work on text summarization has fo- cused on single-document summarization for do- mains such as news (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017; …

Content truncated to 500 words