How Far are We from Robust Long Abstractive Summarization?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2682 - 2698 December 7-11, 2022 ©2022 Association for Computational Linguistics How Far are We from Robust Long Abstractive Summarization? Huan Yee Koh1∗, Jiaxin Ju4∗, He Zhang5, Ming Liu2†, Shirui Pan3† 1Faculty of Information Technology, Monash University, Australia 2School of Information Technology, Deakin University, Australia 3School of Information and Communication Technology, Griffith University, Australia 4Independent Researcher 5Zhongtukexin Co. Ltd., Beijing, China huan.koh@monash.edu, jiaxin.ju.14@gmail.com, zhanghe@kxsz.net m.liu@deakin.edu.au, s.pan@griffith.edu.au Abstract Abstractive summarization has made tremen- dous progress in recent years. In this work, we perform fine-grained human annotations to evaluate long document abstractive summariza- tion systems (i.e., models and metrics) with the aim of implementing them to generate re- liable summaries. For long document abstrac- tive models, we show that the constant strive for state-of-the-art ROUGE results can lead us to generate more relevant summaries but not factual ones. For long document evaluation metrics, human evaluation results show that ROUGE remains the best at evaluating the rele- vancy of a summary. It also reveals important limitations of factuality metrics in detecting dif- ferent types of factual errors and the reasons be- hind the effectiveness of BARTScore. We then suggest promising directions in the endeavor of developing factual consistency metrics. Fi- nally, we release our annotated long document dataset with the hope that it can contribute to the development of metrics across a broader range of summarization settings. 1 Introduction Pre-trained Transformers (Devlin et al., 2019; Raf- fel et al., 2020) have brought tremendous progress in summarizing text in an abstract manner (Rothe et al., 2021). Unlike extractive summarization (Xiao and Carenini, 2019; Cui and Hu, 2021; Ju et al., 2021; Shi et al., 2022), abstractive summa- rization presents a blue-sky potential of generat- ing summaries that are fluent and relevant to the source by intelligently paraphrasing salient con- tents rather than merely copying from source texts (Beltagy et al., 2020; Ju et al., 2020; Zaheer et al., 2020; Huang et al., 2021). Nevertheless, even un- der a short document setting, Transformer-based abstractive models often generate summaries that are repetitive (See et al., 2019; Holtzman et al., ∗ Equal contribution. † Corresponding author. 2019), ungrammatical, and factually inconsistent with the source (Durmus et al., 2020; Kryscinski et al., 2020; Maynez et al., 2020). Furthermore, cur- rent pre-trained Transformers have an input length limit that restricts them to be directly adapted to long document summarization (Lewis et al., 2020; Zhang et al., 2020) as it would lead to a significant loss of salient information in the remaining text. These naturally bring us to a question: How far are we from building a robust abstractive summariza- tion system for long documents? A robust abstractive summarization system should at least have (i) models that can generate high-quality summaries, and (ii) evaluation metrics that can critically assess the relevance and factual- ity of a summary1. However, research on analysis and critiques of models (Wilber et al., 2021; Lad- hak et al., 2022) and metrics (Gabriel …

Content truncated to 500 words