Abstractive Sentence Summarization with Attentive Recurrent Neural Networks

Proceedings of NAACL-HLT 2016, pages 93–98, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Abstractive Sentence Summarization with Attentive Recurrent Neural Networks Sumit Chopra Facebook AI Research spchopra@fb.com Michael Auli Facebook AI Research michaelauli@fb.com Alexander M. Rush Harvard SEAS srush@seas.harvard.edu Abstract Abstractive Sentence Summarization gener- ates a shorter version of a given sentence while attempting to preserve its meaning. We intro- duce a conditional recurrent neural network (RNN) which generates a summary of an in- put sentence. The conditioning is provided by a novel convolutional attention-based encoder which ensures that the decoder focuses on the appropriate input words at each step of genera- tion. Our model relies only on learned features and is easy to train in an end-to-end fashion on large data sets. Our experiments show that the model significantly outperforms the recently proposed state-of-the-art method on the Giga- word corpus while performing competitively on the DUC-2004 shared task. 1 Introduction Generating a condensed version of a passage while preserving its meaning is known as text summariza- tion. Tackling this task is an important step to- wards natural language understanding. Summariza- tion systems can be broadly classified into two cat- egories. Extractive models generate summaries by cropping important segments from the original text and putting them together to form a coherent sum- mary. Abstractive models generate summaries from scratch without being constrained to reuse phrases from the original text. In this paper we propose a novel recurrent neu- ral network for the problem of abstractive sentence summarization. Inspired by the recently proposed architectures for machine translation (Bahdanau et al., 2014), our model consists of a conditional recur- rent neural network, which acts as a decoder to gen- erate the summary of an input sentence, much like a standard recurrent language model. In addition, at every time step the decoder also takes a condition- ing input which is the output of an encoder module. Depending on the current state of the RNN, the en- coder computes scores over the words in the input sentence. These scores can be interpreted as a soft alignment over the input text, informing the decoder which part of the input sentence it should focus on to generate the next word. Both the decoder and en- coder are jointly trained on a data set consisting of sentence-summary pairs. Our model can be seen as an extension of the recently proposed model for the same problem by Rush et al. (2015). While they use a feed-forward neural language model for genera- tion, we use a recurrent neural network. Further- more, our encoder is more sophisticated, in that it explicitly encodes the position information of the in- put words. Lastly, our encoder uses a convolutional network to encode input words. These extensions result in improved performance. The main contribution of this paper is a novel convolutional attention-based conditional recurrent neural network model for the problem of abstractive sentence summarization. Empirically we show that our model beats the state-of-the-art systems of Rush et al. (2015) on …

Content truncated to 500 words