Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4768–4779 June 6–11, 2021. ©2021 Association for Computational Linguistics 4768 Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters Ramakanth Pasunuru1 Mengwen Liu2 Mohit Bansal1 Sujith Ravi2 Markus Dreyer2 1UNC Chapel Hill 2Amazon Alexa {ram,mbansal}@cs.unc.edu, {mengwliu,sujithai,mddreyer}@amazon.com Abstract This paper presents an efficient graph- enhanced approach to multi-document sum- marization (MDS) with an encoder-decoder Transformer model. This model is based on recent advances in pre-training both encoder and decoder on very large text data (Lewis et al., 2019), and it incorporates an efficient encoding mechanism (Beltagy et al., 2020) that avoids the quadratic memory growth typical for traditional Transformers. We show that this powerful combination not only scales to large input documents commonly found when summarizing news clusters; it also enables us to process additional input in the form of auxiliary graph representations, which we derive from the multi-document clusters. We present a mechanism to incorporate such graph information into the encoder-decoder model that was pre-trained on text only. Our approach leads to significant improvements on the Multi-News dataset, overall leading to an average 1.8 ROUGE score improvement over previous work (Li et al., 2020). We also show improvements in a transfer-only setup on the DUC-2004 dataset. The graph encodings lead to summaries that are more abstractive. Human evaluation shows that they are also more informative and factually more consistent with their input documents.1 1 Introduction Abstractive Multi-Document Summarization (MDS), the task of writing a consolidated sum- mary of the main information from multiple documents, has seen advancements with the introduction of large-scale datasets and powerful Transformer-based models (Liu et al., 2018; Liu and Lapata, 2019; Fabbri et al., 2019). However, some of the key challenges of MDS include lack of proper inter-document context-aware information, improper logical flow of information, and need 1All our code publicly available at: https://github. com/amazon-research/BartGraphSumm. Decoder Cluster texts Truncated concatenated text </d> </d> Linearized graph text Summary Graph Encoder (linear attention) Text Encoder (linear attention) Figure 1: Illustration of our dual-encoder approach to summarizing multi-document clusters with graph en- codings. The truncated concatenated text contains the beginnings of each cluster document; the graphs con- tain information from the full documents. for external deep context representations. Liu and Lapata (2019) and Li et al. (2020) have addressed the inter-document context modeling to some extent with local and global attention, and document-level similarity graphs. Further, Li et al. (2020) have addressed the later part of using external contextual information (large pre-trained language models, e.g., RoBERTa (Liu et al., 2019)) to improve the performance of MDS models. However, these pre-trained language models are (1) not scalable for long documents because of their encoding length limit and quadratic memory growth; and (2) they do not jointly explore alternate auxiliary information, e.g., semantic graphs derived from multi-document clusters. Addressing these issues, we present an efficient graph-enhanced approach to multi-document sum- marization using a pre-trained encoder-decoder Transformer model (Lewis et al., 2019), …

Content truncated to 500 words