A Neural Attention Model for Abstractive Sentence Summarization

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A Neural Attention Model for Sentence Summarization Alexander M. Rush Facebook AI Research / Harvard SEAS srush@seas.harvard.edu Sumit Chopra Facebook AI Research spchopra@fb.com Jason Weston Facebook AI Research jase@fb.com Abstract Summarization based on text extraction is inherently limited, but generation-style ab- stractive methods have proven challeng- ing to build. In this work, we propose a fully data-driven approach to abstrac- tive sentence summarization. Our method utilizes a local attention-based model that generates each word of the summary con- ditioned on the input sentence. While the model is structurally simple, it can eas- ily be trained end-to-end and scales to a large amount of training data. The model shows significant performance gains on the DUC-2004 shared task compared with several strong baselines. 1 Introduction Summarization is an important challenge of natu- ral language understanding. The aim is to produce a condensed representation of an input text that captures the core meaning of the original. Most successful summarization systems utilize extrac- tive approaches that crop out and stitch together portions of the text to produce a condensed ver- sion. In contrast, abstractive summarization at- tempts to produce a bottom-up summary, aspects of which may not appear as part of the original. We focus on the task of sentence-level sum- marization. While much work on this task has looked at deletion-based sentence compression techniques (Knight and Marcu (2002), among many others), studies of human summarizers show that it is common to apply various other operations while condensing, such as paraphrasing, general- ization, and reordering (Jing, 2002). Past work has modeled this abstractive summarization prob- lem either using linguistically-inspired constraints (Dorr et al., 2003; Zajic et al., 2004) or with syn- tactic transformations of the input text (Cohn and Figure 1: Example output of the attention-based summa- rization (ABS) system. The heatmap represents a soft align- ment between the input (right) and the generated summary (top). The columns represent the distribution over the input after generating each word. Lapata, 2008; Woodsend et al., 2010). These ap- proaches are described in more detail in Section 6. We instead explore a fully data-driven approach for generating abstractive summaries. Inspired by the recent success of neural machine translation, we combine a neural language model with a con- textual input encoder. Our encoder is modeled off of the attention-based encoder of Bahdanau et al. (2014) in that it learns a latent soft alignment over the input text to help inform the summary (as shown in Figure 1). Crucially both the encoder and the generation model are trained jointly on the sentence summarization task. The model is de- scribed in detail in Section 3. Our model also in- corporates a beam-search decoder as well as addi- tional features to model extractive elements; these aspects are discussed in Sections 4 and 5. This approach to summarization, which we call Attention-Based Summarization (ABS), incorpo- rates less linguistic …

Content truncated to 500 words