On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9308–9319, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9308 On Extractive and Abstractive Neural Document Summarization with Transformer Language Models Jonathan Pilault∗ 1,2,3, Raymond Li∗1, Sandeep Subramanian∗1,2,4 and Christopher Pal1,2,3,4,5 1Element AI 2Mila, 3Polytechnique Montreal, 4University of Montreal, 5Canada CIFAR AI Chair firstname.lastname@elementai.com Abstract We present a method to produce abstractive summaries of long documents that exceed sev- eral thousand words via neural abstractive summarization. We perform a simple extrac- tive step before generating a summary, which is then used to condition the transformer lan- guage model on relevant information before being tasked with generating a summary. We also show that this approach produces more ab- stractive summaries compared to prior work that employs a copy mechanism while still achieving higher ROUGE scores. We pro- vide extensive comparisons with strong base- line methods, prior state of the art work as well as multiple variants of our approach in- cluding those using only transformers, only extractive techniques and combinations of the two. We examine these models using four dif- ferent summarization tasks and datasets: arXiv papers, PubMed papers, the Newsroom and BigPatent datasets. We find that transformer based methods produce summaries with fewer n-gram copies, leading to n-gram copying statistics that are more similar to human gener- ated abstracts. We include a human evaluation, finding that transformers are ranked highly for coherence and fluency, but purely extractive methods score higher for informativeness and relevance. We hope that these architectures and experiments may serve as strong points of comparison for future work.1 1 Introduction Automatic text summarization is the process of compressing a document while preserving key in- formation content and meaning. This process is often achieved through extractive or abstractive ∗Authors contributed equally to this work 1Note: The abstract above was collaboratively written by the authors and one of the models presented in this paper based on an earlier draft of this paper. Neural Document Summarization with Sentence Pointer Networks and Transformer Language Models Sandeep Subramanian1,2,3,⇤, Raymond Li1,⇤, Jonathan Pilault1,2,4,⇤, Christopher Pal1,2,4,5 1Element AI, 2Montréal Institute for Learning Algorithms, 3Université de Montréal, 4École Polytechnique de Montréal, 5Canada CIFAR AI Chair 1{jonathan.pilault}@elementai.com Abstract We demonstrate that Transformer language models are extremely promising at summarizing long texts, and provide a new approach to deep summarization that can be used to generate more "abstractive" summaries. We show that our approach produces more abstractive summaries than state-of-the-art methods without a copy mechanism. We provide an application to text summarization of the arXiv and PubMed datasets, and show that our model outperforms other popular summa- rization techniques. We also discuss a simple neural extractive model based on pointers networks trained on documents and their salient sentences. We show that this model can be used to augment Transformer language models to generate better summarization results. Note: The abstract above was generated by one of the models presented in this paper, as a summary of this paper. 1 Introduction Language models (LMs) are trained …

Content truncated to 500 words