CFSum Coarse-to-Fine Contribution Network for Multimodal Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8538–8553 July 9-14, 2023 ©2023 Association for Computational Linguistics CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarization Min Xiao 1,2, Junnan Zhu1,2, Haitao Lin1,2, Yu Zhou1,3∗, Chengqing Zong1,2 1 State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3 Fanyu AI Laboratory, Zhongke Fanyu Technology Co., Ltd, Beijing, China {min.xiao, junnan.zhu, haitao.lin, yzhou, cqzong}@nlpr.ia.ac.cn, Abstract Multimodal summarization usually suffers from the problem that the contribution of the visual modality is unclear. Existing multimodal summarization approaches focus on design- ing the fusion methods of different modalities, while ignoring the adaptive conditions under which visual modalities are useful. Therefore, we propose a novel Coarse-to-Fine contribu- tion network for multimodal Summarization (CFSum) to consider different contributions of images for summarization. First, to eliminate the interference of useless images, we propose a pre-filter module to abandon useless images. Second, to make accurate use of useful images, we propose two levels of visual complement modules, word level and phrase level. Specif- ically, image contributions are calculated and are adopted to guide the attention of both tex- tual and visual modalities. Experimental results have shown that CFSum significantly outper- forms multiple strong baselines on the standard benchmark. Furthermore, the analysis verifies that useful images can even help generate non- visual words which are implicitly represented in the image1. 1 Introduction With the information explosion, the internet is flooded with various multimodal information. Mul- timodal summarization (MMS) can help generate more abundant and comprehensive summary infor- mation than unimodal based on extra visual infor- mation. Existing studies on multimodal summariza- tion include multimodal sentence summarization (Li et al., 2018b), multimodal summarization with ∗Corresponding author. 1Code is available at https://github.com/ xiaomin418/CFSum 44 45 46 47 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ROUGE-1 Masking Rate No-masked (Concat) Image-masked (Concat) No-masked (Attn) Image-masked (Attn) Figure 1: Experiments on existing mainstream multi- modal summarization models. The performance is not affected by masking images. “Concat” is the concate- nate fusion method, and “Attn” is the attention-based fusion method. multimodal output (Zhu et al., 2018), multimodal meeting summarization (Li et al., 2019) and so on. In this paper, we focus on the task that generat- ing a text summary based on the input of a text and an image. It has been proved that integrating multimodal data can help improve the quality of the summary (Li et al., 2018b; Jangra et al., 2020; Palaskar et al., 2019; Yu et al., 2021). However, it is unclear whether the visual modal- ity can indeed benefit the process of summariza- tion. Thus, we conduct an experiment to explore the influence of masking images on the summary. As shown in Figure 1, the solid lines mean the performance of summary generated by masking portions of images, and the dashed lines indicate the origin performance. It can be …

Content truncated to 500 words