Evaluating and Improving Factuality in Multimodal Abstractive Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9632 - 9648 December 7-11, 2022 ©2022 Association for Computational Linguistics Evaluating and Improving Factuality in Multimodal Abstractive Summarization David Wan and Mohit Bansal University of North Carolina at Chapel Hill {davidwan,mbansal}@cs.unc.edu Abstract Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judg- ment, but they do not account for the vision modality and thus are not adequate for vision- and-language summarization. We propose CLIPBERTSCORE, a simple weighted com- bination of CLIPScore (Hessel et al., 2021) and BERTScore (Zhang* et al., 2020) to lever- age the robustness and strong factuality detec- tion performance between image-summary and document-summary, respectively. Next, due to the lack of meta-evaluation benchmarks to evaluate the quality of multimodal factuality metrics, we collect human judgments of fac- tuality with respect to documents and images. We show that this simple combination of two metrics in the zero-shot setting achieves higher correlations than existing factuality metrics for document summarization, outperforms an ex- isting multimodal summarization metric, and performs competitively with strong multimodal factuality metrics specifically fine-tuned for the task. Our thorough analysis demonstrates the robustness and high correlation of CLIP- BERTSCORE and its components on four fac- tuality metric-evaluation benchmarks. Finally, we demonstrate two practical downstream ap- plications of our CLIPBERTSCORE metric: for selecting important images to focus on dur- ing training, and as a reward for reinforcement learning to improve factuality of multimodal summary generation w.r.t automatic and human evaluation.1 1 Introduction Multimodal abstractive summarization is the task of generating an abridged text that contains the most important information of the source inputs from various modalities. This challenging task builds upon the success of document summariza- tion, where the input is only text documents. For 1Our data and code are publicly available at https:// github.com/meetdavidwan/faithful-multimodal-summ document summarization, there has been tremen- dous progress in improving the quality of the sum- maries with the help of large pre-trained models (Lewis et al., 2020; Zhang et al., 2020; Raffel et al., 2020). However, one crucial problem for such models is hallucination, where the model gener- ates contents that are not present or entailed by the document (Maynez et al., 2020; Falke et al., 2019). While there have been significant advancements in developing metrics that correlate highly with the human judgment of factuality (Kryscinski et al., 2020; Durmus et al., 2020; Goyal and Durrett, 2021; Scialom et al., 2021), these metrics only measure factuality between the document and the summary. The lack of judgment between other modalities, such as vision, and the summary makes such metrics not suitable for multimodal settings. We demonstrate this with the example in Figure 1. The given summary captures less relevant informa- tion (cutting the nail) from the document, but it is still considered factual to the document. However, the image shows the main point of the document (finding the place where the nail separates from the quick), making the summary not factual with respect to the …

Content truncated to 500 words