Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3995–4007 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3995 Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization Tiezheng Yu∗, Wenliang Dai∗, Zihan Liu, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {tyuah,wdaiai,zliucr}@connect.ust.hk, pascale@ece.ust.hk Abstract Multimodal abstractive summarization (MAS) models that summarize videos (vision modal- ity) and their corresponding transcripts (text modality) are able to extract the essential infor- mation from massive multimodal data on the Internet. Recently, large-scale generative pre- trained language models (GPLMs) have been shown to be effective in text generation tasks. However, existing MAS models cannot lever- age GPLMs’ powerful generation ability. To fill this research gap, we aim to study two re- search questions: 1) how to inject visual in- formation into GPLMs without hurting their generation ability; and 2) where is the opti- mal place in GPLMs to inject the visual in- formation? In this paper, we present a sim- ple yet effective method to construct vision guided (VG) GPLMs for the MAS task using attention-based add-on layers to incorporate vi- sual information while maintaining their orig- inal text generation ability. Results show that our best model significantly surpasses the prior state-of-the-art model by 5.7 ROUGE-1, 5.3 ROUGE-2, and 5.1 ROUGE-L scores on the How2 dataset (Sanabria et al., 2018), and our visual guidance method contributes 83.6% of the overall improvement. Furthermore, we con- duct thorough ablation studies to analyze the ef- fectiveness of various modality fusion methods and fusion locations. 1 1 Introduction Multimodal abstractive summarization (MAS) aims to take advantage of data from multiple modalities and provides a short, concise and readable textual summary to let users quickly acquire their essential information (Sanabria et al., 2018; Palaskar et al., 2019; Liu et al., 2020). MAS has become an increasingly popular research area thanks to the proliferation of online multimedia content and the increasing availability of multimodal data. ∗∗ The two authors contribute equally. 1The code is available at: https://github.com/ HLTCHKUST/VG-GPLMs Video Frames Transcript: so now we are going to go over some basics sheet music readings for the key of g flat major. so you noticed the key of g flat, when you are reading real books, there is going to be a treble cleft here. it is going to have 6 flats 1, b flat, e flat, a flat, d flat, g flat and c flat. so 6 flats equals key of g flat. [...] so if you have a flat and there is a natural sign, play the a. so go through the scale and you've got g flat, a flat, d flat, c, flat, d flat, e flat and f, so f is your only 9 flat note in the scale. (No mention of the piano) Reference Summary: learn how to read and write music intervals for improving your playing and improvisational skills on the …

Content truncated to 500 words