Extractive Summarization of Long Documents by Combining Global and Local Context

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3011–3021, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3011 Extractive Summarization of Long Documents by Combining Global and Local Context Wen Xiao and Giuseppe Carenini Department of Computer Science University of British Columbia Vancouver, BC, Canada, V6T 1Z4 {xiaowen3, carenini}@cs.ubc.ca Abstract In this paper, we propose a novel neural single- document extractive summarization model for long documents, incorporating both the global context of the whole document and the local context within the current topic. We evalu- ate the model on two datasets of scientific pa- pers , Pubmed and arXiv, where it outperforms previous work, both extractive and abstractive models, on ROUGE-1, ROUGE-2 and ME- TEOR scores. We also show that, consistently with our goal, the benefits of our method be- come stronger as we apply it to longer docu- ments. Rather surprisingly, an ablation study indicates that the benefits of our model seem to come exclusively from modeling the local context, even for the longest documents. 1 Introduction Single-document summarization is the task of generating a short summary for a given document. Ideally, the generated summaries should be fluent and coherent, and should faithfully maintain the most important information in the source docu- ment. This is a very challenging task, because it arguably requires an in-depth understanding of the source document, and current automatic solutions are still far from human performance (Allahyari et al., 2017).VII Single-document summarization can be either extractive or abstractive. Extractive methods typ- ically pick sentences directly from the original document based on their importance, and form the summary as an aggregate of these sentences. Usually, summaries generated in this way have a better performance on fluency and grammar, but they may contain much redundancy and lack in coherence across sentences. In contrast, abstrac- tive methods attempt to mimic what humans do VIISentence coloring and Roman numbering will be ex- plained in the result sub-section 4.5. by first extracting content from the source docu- ment and then produce new sentences that aggre- gate and organize the extracted information. Since the sentences are generated from scratch they tend to have a relatively worse performance on flu- ency and grammar. Furthermore, while abstractive summaries are typically less redundant, they may end up including misleading or even utterly false statements, because the methods to extract and ag- gregate information form the source document are still rather noisy. In this work, we focus on extracting informative sentences from a given document (without dealing with redundancy), especially when the document is relatively long (e.g., scientific articles). Most recent works on neural extractive summa- rization have been rather successful in generating summaries of short news documents (around 650 words/document) (Nallapati et al., 2016) by ap- plying neural Seq2Seq models (Cheng and Lap- ata, 2016). However when it comes to long doc- uments, these models tend to struggle with longer sequences because at each decoding step, …

Content truncated to 500 words