SummN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1592 - 1604 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SUMMN: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents Yusen Zhang♣ Ansong Ni† Ziming Mao† Chen Henry Wu ‡ Chenguang Zhu♦ Budhaditya Deb♦ Ahmed H. Awadallah♦ Dragomir Radev† Rui Zhang♣ ♣ Penn State University ‡ Carnegie Mellon University† Yale University ♦ Microsoft Research {yfz5488,rmz5227}@psu.edu, {ansong.ni,dragomir.radev}@yale.edu Abstract Text summarization helps readers capture salient information from documents, news, in- terviews, and meetings. However, most state- of-the-art pretrained language models (LM) are unable to efficiently process long text for many summarization tasks. In this pa- per, we propose SUMMN, a simple, flexi- ble, and effective multi-stage framework for input texts that are longer than the maxi- mum context length of typical pretrained LMs. SUMMN first splits the data samples and gener- ates a coarse summary in multiple stages and then produces the final fine-grained summary based on it. Our framework can process in- put text of arbitrary length by adjusting the number of stages, while keeping the LM in- put size fixed. Moreover, it can deal with both single-source documents and dialogues, and it can be used on top of different back- bone abstractive summarization models. To the best of our knowledge, SUMMN is the first multi-stage split-then-summarize frame- work for long input summarization. Our ex- periments demonstrate that SUMMN outper- forms previous state-of-the-art methods by im- proving ROUGE scores on three long meet- ing summarization datasets AMI, ICSI, and QMSum, two long TV series datasets from SummScreen, and a long document summa- rization dataset GovReport. Our data and code are available at https://github.com/ psunlpgroup/Summ-N. 1 Introduction Abstractive summarization helps readers capture salient information from various sources such as documents, news, interviews, and meetings. Pre- vious work has primarily focused on short texts of news (Gehrmann et al., 2018; Zhang et al., 2019) and short conversations (Gliwa et al., 2019; Chen and Yang, 2021). Recently proposed longer dia- logue and document summarization tasks (Zhong et al., 2021b; Huang et al., 2021; Chen et al., 2021; Zhu et al., 2021a) pose challenges for current large pretrained language models due to the time and memory complexity of training, as well as limited input lengths these models can consume. A common method to handle long text re- duces the input to a shorter one. This can be accomplished by truncating inputs (Lewis et al., 2020) or employing retrieve-then-summarize pipelines (Zhong et al., 2021b). However, these methods break the dependency of the context and decrease the number of tokens that the model can read, i.e., the receptive field of the model. The cutting-off model depends on the lead bias of the source text, while the retrieve-then-summarize models heavily rely on the independence of re- trieved units (turns or sentences) which are usually scattered throughout the source text. Another approach optimizes the attention mech- anism in Transformers to accommodate longer in- puts by reducing the impact of quadratic …

Content truncated to 500 words