Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 280–290, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond Ramesh Nallapati IBM Watson nallapati@us.ibm.com Bowen Zhou IBM Watson zhou@us.ibm.com Cicero dos Santos IBM Watson cicerons@us.ibm.com Ça˘glar G˙ulçehre Université de Montréal gulcehrc@iro.umontreal.ca Bing Xiang IBM Watson bingxia@us.ibm.com Abstract In this work, we model abstractive text summarization using Attentional Encoder- Decoder Recurrent Neural Networks, and show that they achieve state-of-the-art per- formance on two different corpora. We propose several novel models that address critical problems in summarization that are not adequately modeled by the basic architecture, such as modeling key-words, capturing the hierarchy of sentence-to- word structure, and emitting words that are rare or unseen at training time. Our work shows that many of our proposed models contribute to further improvement in performance. We also propose a new dataset consisting of multi-sentence sum- maries, and establish performance bench- marks for further research. 1 Introduction Abstractive text summarization is the task of gen- erating a headline or a short summary consisting of a few sentences that captures the salient ideas of an article or a passage. We use the adjective ‘ab- stractive’ to denote a summary that is not a mere selection of a few existing passages or sentences extracted from the source, but a compressed para- phrasing of the main contents of the document, potentially using vocabulary unseen in the source document. This task can also be naturally cast as map- ping an input sequence of words in a source doc- ument to a target sequence of words called sum- mary. In the recent past, deep-learning based mod- els that map an input sequence into another out- put sequence, called sequence-to-sequence mod- els, have been successful in many problems such as machine translation (Bahdanau et al., 2014), speech recognition (Bahdanau et al., 2015) and video captioning (Venugopalan et al., 2015). In the framework of sequence-to-sequence models, a very relevant model to our task is the atten- tional Recurrent Neural Network (RNN) encoder- decoder model proposed in Bahdanau et al. (2014), which has produced state-of-the-art per- formance in machine translation (MT), which is also a natural language task. Despite the similarities, abstractive summariza- tion is a very different problem from MT. Unlike in MT, the target (summary) is typically very short and does not depend very much on the length of the source (document) in summarization. Addi- tionally, a key challenge in summarization is to op- timally compress the original document in a lossy manner such that the key concepts in the original document are preserved, whereas in MT, the trans- lation is expected to be loss-less. In translation, there is a strong notion of almost one-to-one word- level alignment between source and target, but in summarization, it is less obvious. We make the following main contributions in this work: (i) We apply the off-the-shelf atten- tional encoder-decoder RNN that was originally developed for machine translation to summariza- tion, and …

Content truncated to 500 words