MM-AVS: A Full-Scale Dataset for Multi-modal Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5922–5926 June 6–11, 2021. ©2021 Association for Computational Linguistics 5922 MM-AVS: A Full-Scale Dataset for Multi-modal Summarization Xiyan Fu1, Jun Wang2, Zhenglu Yang1 1Nankai University, China 2 Ludong University, China {fuxiyan,junwang}@mail.nankai.edu.cn, yangzl@nankai.edu.cn Abstract Multimodal summarization becomes increas- ingly significant as it is the basis for question answering, Web search, and many other down- stream tasks. However, its learning materials have been lacking a holistic organization by integrating resources from various modalities, thereby lagging behind the research progress of this field. In this study, we present a full-scale multimodal dataset comprehensively gathering documents, summaries, images, cap- tions, videos, audios, transcripts, and titles in English from CNN and Daily Mail. To our best knowledge, this is the first collection that spans all modalities and nearly comprises all types of materials available in this commu- nity. In addition, we devise a baseline model based on the novel dataset, which employs a newly proposed Jump-Attention mechanism based on transcripts. The experimental results validate the important assistance role of the ex- ternal information for multimodal summariza- tion. 1 Introduction Multimodal summarization refines salient infor- mation from one or more modalities, including text, image, audio, and video ones (Evangelopou- los et al., 2013; Li et al., 2017). Given the rapid dissemination of multimedia data over the Inter- net, multimodal summarization has been widely explored in recent years. Meanwhile, some mul- timodal datasets (Li et al., 2017; Zhu et al., 2018; Sanabria et al., 2018; Li et al., 2020a) have been introduced to advance the development of this re- search field. However, a majority of them are re- stricted in scale and too oriented, such as being less than one hundred examples or merely containing Chinese texts. Moreover, the materials from differ- ent modalities are rarely collected across the board, especially videos and their accompanying materi- als that possess abundant external information for multimodal comprehension and fusion. In this work, we introduce a full-scale Multi- modal Article and Video Summarization (MM- AVS) dataset 1 with documents, summaries, im- ages, captions, videos, audios, transcripts, and titles in English. The significance of MM-AVS for the multimodal summarization community includes but not limited to: 1) MM-AVS is a large-scale mul- timodal collection compared with existing video containing dataset and its generation codes 1 has been released, which can be readily extended for existing and future multimodal summarization ap- proaches; 2) MM-AVS is collected from CNN2 and Daily Mail3, which makes it available to more researchers due to English-based and comparable with the popular text-based CNN/Daily Mail cor- pus; and 3) MM-AVS firstly collects nearly all types of materials from all modalities, inclusively with videos, audios, transcripts, images, captions, and titles that are rarely assembled. In addition, we implement a general multimodal summarization baseline based on transcripts for multimodal summarization on MM-AVS. This method employs a Jump-Attention mechanism to align features between text and video. Further, we use the multi-task learning …

Content truncated to 500 words