Neural Paraphrase Generation with Stacked Residual LSTM Networks

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2923–2934, Osaka, Japan, December 11-17 2016. Neural Paraphrase Generation with Stacked Residual LSTM Networks Aaditya Prakash1,2, Sadid A. Hasan2, Kathy Lee2, Vivek Datla2, Ashequl Qadir2, Joey Liu2, Oladimeji Farri2 1Brandeis University, Waltham, MA, USA 2Artificial Intelligence Laboratory, Philips Research North America, Cambridge, MA, USA {aprakash,aaditya.prakash}@{brandeis.edu,philips.com} {sadid.hasan,kathy.lee 1,vivek.datla}@philips.com {ashequl.qadir,joey.liu,dimeji.farri}@philips.com Abstract In this paper, we propose a novel neural approach for paraphrase generation. Conventional para- phrase generation methods either leverage hand-written rules and thesauri-based alignments, or use statistical machine learning principles. To the best of our knowledge, this work is the first to explore deep learning models for paraphrase generation. Our primary contribution is a stacked residual LSTM network, where we add residual connections between LSTM layers. This allows for efficient training of deep LSTMs. We evaluate our model and other state-of-the-art deep learning models on three different datasets: PPDB, WikiAnswers, and MSCOCO. Evaluation results demonstrate that our model outperforms sequence to sequence, attention-based, and bi- directional LSTM models on BLEU, METEOR, TER, and an embedding-based sentence similarity metric. 1 Introduction Paraphrasing, the act to express the same meaning in different possible ways, is an important subtask in various Natural Language Processing (NLP) applications such as question answering, information extraction, information retrieval, summarization and natural language generation. Research on para- phrasing methods typically aims at solving three related problems: (1) recognition (i.e. to identify if two textual units are paraphrases of each other), (2) extraction (i.e. to extract paraphrase instances from a thesaurus or a corpus), and (3) generation (i.e. to generate a reference paraphrase given a source text) (Madnani and Dorr, 2010). In this paper, we focus on the paraphrase generation problem. Paraphrase generation has been used to gain performance improvements in several NLP applications, for example, by generating query variants or pattern alternatives for information retrieval, information extraction or question answering systems, by creating reference paraphrases for automatic evaluation of machine translation and document summarization systems, and by generating concise or simplified information for sentence compression or sentence simplification systems (Madnani and Dorr, 2010). Traditional paraphrase generation methods exploit hand-crafted rules (McKeown, 1983) or automatically learned complex paraphrase patterns (Zhao et al., 2009), use thesaurus-based (Hassan et al., 2007) or semantic analysis driven natural language generation approaches (Kozlowski et al., 2003), or leverage statistical machine learning theory (Quirk et al., 2004; Wubben et al., 2010). In this paper, we propose to use deep learning principles to address the paraphrase generation problem. Recently, techniques like sequence to sequence learning (Sutskever et al., 2014) have been applied to various NLP tasks with promising results, for example, in the areas of machine translation (Cho et al., 2014; Bahdanau et al., 2015), speech recognition (Li and Wu, 2015), language modeling (Vinyals et al., 2015), and dialogue systems (Serban et al., 2016). Although paraphrase generation can be formulated as a sequence to sequence learning task, not much work has been done in this area with regard to applica- tions of state-of-the-art …

Content truncated to 500 words