Multimodal Sentence Summarization via Multimodal Selective Encoding

Proceedings of the 28th International Conference on Computational Linguistics, pages 5655–5667 Barcelona, Spain (Online), December 8-13, 2020 5655 Multimodal Sentence Summarization via Multimodal Selective Encoding Haoran Li1, Junnan Zhu2,3, Jiajun Zhang2,3,4, Chengqing Zong2,3,5, Xiaodong He1 1 JD AI Research 2National Laboratory of Pattern Recognition, Institute of Automation, CAS 3 University of Chinese Academy of Sciences 4 Beijing Academy of Artificial Intelligence 5 CAS Center for Excellence in Brain Science and Intelligence Technology {lihaoran24, xiaodong.he}@jd.com {junnan.zhu, jjzhang, cqzong}@nlpr.ia.ac.cn Abstract This paper studies the problem of generating a summary for a given sentence-image pair. Ex- isting multimodal sequence-to-sequence approaches mainly focus on enhancing the decoder by visual signals, while ignoring that the image can improve the ability of the encoder to identify highlights of a news event or a document. Thus, we propose a multimodal selective gate network that considers reciprocal relationships between textual and multi-level visual features, includ- ing global image descriptor, activation grids, and object proposals, to select highlights of the event when encoding the source sentence. In addition, we introduce a modality regularization to encourage the summary to capture the highlights embedded in the image more accurately. To verify the generalization of our model, we adopt the multimodal selective gate to the text-based decoder and multimodal-based decoder. Experimental results on a public multimodal sentence summarization dataset demonstrate the advantage of our models over baselines. Further analysis suggests that our proposed multimodal selective gate network can effectively select important information in the input sentence. 1 Introduction Text summarization is a task that condenses a long sentence to a short version. Existing researches (Rush et al., 2015; Chopra et al., 2016; Zeng et al., 2016; Li et al., 2017; Tan et al., 2017; Zhou et al., 2017; Zhang et al., 2018; Li et al., 2020b) produce summary only from the text. However, it has been proved that human understands the text relying on multimodal information (Waltz, 1980; Srihari, 1994; He and Deng, 2017), such as linguistic and visual signals. In this paper, we focus on the multimodal summa- rization task (Li et al., 2018a) that generates summary simultaneously drawing knowledge from coupled text and image, which can facilitate other applications such as image captioning (Xu et al., 2015; Vinyals et al., 2015), multimodal news summarization (Narayan et al., 2017; Zhu et al., 2018; Chen and Zhuge, 2018), and electronic commerce (e-commerce) product description generation (Chen et al., 2019; Elad et al., 2019; Zhang et al., 2019; Li et al., 2020a), etc. Intuitively, it is easier for a reader to grasp the highlight of a news event by viewing an image than by reading a long text. Hence we believe that the image will benefit text summarization system. Figure 1 il- lustrates this phenomenon. For a given source sentence, a paired image visualizes a set of event highlight words, which highly correlates with the reference summary. Multimodal sequence-to-sequence (seq2seq) learning has been widely explored in machine transla- tion (MT) (Calixto et al., 2017; Caglayan et al., 2017; Helcl et al., 2018; Gr¨onroos …

Content truncated to 500 words