MLASK: Multimodal Summarization of Video-based News Articles

Findings of the Association for Computational Linguistics: EACL 2023, pages 910–924 May 2-6, 2023 ©2023 Association for Computational Linguistics MLASK: Multimodal Summarization of Video-based News Articles Mateusz Krubi´nski and Pavel Pecina Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics {krubinski,pecina}@ufal.mff.cuni.cz Abstract In recent years, the pattern of news consump- tion has been changing. The most popular mul- timedia news formats are now multimodal – the reader is often presented not only with a textual article but also with a short, vivid video. To draw the attention of the reader, such video- based articles are usually presented as a short textual summary paired with an image thumb- nail. In this paper, we introduce MLASK1 (MultimodaL Article Summarization Kit) – a new dataset of video-based news articles paired with a textual summary and a cover pic- ture, all obtained by automatically crawling several news websites. We demonstrate how the proposed dataset can be used to model the task of multimodal summarization by training a Transformer-based neural model. We also examine the effects of pre-training when the us- age of generative pre-trained language models helps to improve the model performance, but (additional) pre-training on the simpler task of text summarization yields even better results. Our experiments suggest that the benefits of pre-training and using additional modalities in the input are not orthogonal. 1 Introduction Automatic summarization is one of the basic tasks both in Natural Language Processing – text summa- rization – and in Computer Vision – video summa- rization. Multimodal summarization (MMS) builds a bridge between those two fields. Early works on multimodal summarization ex- plored the usage of the secondary modality as an auxiliary source of information to guide the refine- ment process of the main modality. Li et al. (2017) collected videos and news articles covering a hand- crafted list of recent significant world events by querying a web search engine and trained a model to mimic the reference summaries written by hu- man annotators. Zhu et al. (2018) were the first to 1https://github.com/ufal/MLASK introduce the task of Multimodal Summarization with Multimodal Output (MSMO). They collected a large-scale dataset of news articles paired with corresponding images and trained a system to gen- erate a textual summary and choose a single image as a pictorial summary. By introducing the multi- modal output, a uni-modal solution was no longer sufficient as a baseline. Building upon this, Li et al. (2020b) extended the task to video-based MSMO. Based on a textual document and a short video clip, besides generating the textual summary, the system was also challenged to select a single frame from the video as a cover picture. We believe there is still a lot of questions that remain unanswered, e.g.: How to evaluate multi- modal outputs? or How to approach pre-training? In this paper, we contribute to the area of video- based MSMO (VMSMO) by: 1) introducing a full- scale VMSMO dataset in Czech, extending the very limited available resources for this task to a new …

Content truncated to 500 words