Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2934–2951 July 9-14, 2023 ©2023 Association for Computational Linguistics Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization Yunlong Liang1∗, Fandong Meng2, Jinan Xu1†, Jiaan Wang2, Yufeng Chen1 and Jie Zhou2 1Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc, China {yunlongliang,jaxu}@bjtu.edu.cn fandongmeng@tencent.com Abstract Multimodal abstractive summarization (MAS) aims to produce a concise summary given the multimodal data (text and vision). Existing studies mainly focus on how to effectively use the visual features from the perspective of an article, having achieved impressive success on the high-resource English dataset. However, less attention has been paid to the visual fea- tures from the perspective of the summary, which may limit the model performance, es- pecially in the low- and zero-resource scenar- ios. In this paper, we propose to improve the summary quality through summary-oriented vi- sual features. To this end, we devise two auxil- iary tasks including vision to summary task and masked image modeling task. Together with the main summarization task, we optimize the MAS model via the training objectives of all these tasks. By these means, the MAS model can be enhanced by capturing the summary- oriented visual features, thereby yielding more accurate summaries. Experiments on 44 lan- guages, covering mid-high-, low-, and zero- resource scenarios, verify the effectiveness and superiority of the proposed approach, which achieves state-of-the-art performance under all scenarios. Additionally, we will contribute a large-scale multilingual multimodal abstractive summarization (MM-Sum) dataset.1 1 Introduction Given an article and several images as inputs, as shown in Fig. 1, multimodal abstractive summa- rization (MAS) (Sanabria et al., 2018; Li et al., 2017, 2018a; Zhu et al., 2018; Jangra et al., 2020) aims to generate a concise textual summary, which can help people quickly grasp the core information. Therefore, MAS has widespread application and ∗Work was done when Liang and Wang was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Jinan Xu is the corresponding author. 1The code and data are publicly available at: https:// github.com/XL2248/SOV-MAS. Figure 1: An example of our MM-Sum dataset. Inputs: an article and image sequence pair; Output: summary. As we can see, the image sequence also concisely para- phrases the summary. The red content indicates its asso- ciated object is useless to the summary while the green counterparts represent important information. attracts increasing attention with the rapid prolif- eration of multimedia content (Apostolidis et al., 2021; Feng et al., 2022; Qiu et al., 2022). Recently, many studies have been carried out to effectively inject the visual features into MAS mod- els (Li et al., 2018b, 2020b; Zhu et al., 2020, 2021; Zhang et al., 2021b,a; Palaskar et al., 2019; Liu et al., 2020; Yu et al., 2021a). For instance, Palaskar et al. (2019) and Zhang et al. (2021a) explore the hierarchy between the textual article and visual fea- tures, and integrate them into the MAS model. Liu …

Content truncated to 500 words