Sentence Compression by Deletion with LSTMs

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 360–368, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Sentence Compression by Deletion with LSTMs Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, Oriol Vinyals Google Research {katjaf,ealfonseca,crcarlos,lukaszkaiser,vinyals}@google.com Abstract We present an LSTM approach to deletion-based sentence compression where the task is to translate a sentence into a sequence of zeros and ones, cor- responding to token deletion decisions. We demonstrate that even the most basic version of the system, which is given no syntactic information (no PoS or NE tags, or dependencies) or desired compression length, performs surprisingly well: around 30% of the compressions from a large test set could be regenerated. We compare the LSTM system with a competitive baseline which is trained on the same amount of data but is additionally provided with all kinds of linguistic features. In an experiment with human raters the LSTM- based model outperforms the baseline achieving 4.5 in readability and 3.8 in informativeness. 1 Introduction Sentence compression is a standard NLP task where the goal is to generate a shorter paraphrase of a sentence. Dozens of systems have been intro- duced in the past two decades and most of them are deletion-based: generated compressions are token subsequences of the input sentences (Jing, 2000; Knight & Marcu, 2000; McDonald, 2006; Clarke & Lapata, 2008; Berg-Kirkpatrick et al., 2011, to name a few). Existing compression systems heavily use syn- tactic information to minimize chances of intro- ducing grammatical mistakes in the output. common approach is to use only some syntacticA information (Jing, 2000; Clarke & Lapata, 2008, among others) or use syntactic features as signals in a statistical model (McDonald, 2006). It is prob- ably even more common to operate on syntactic trees directly (dependency or constituency) and generate compressions by pruning them (Knight & Marcu, 2000; Berg-Kirkpatrick et al., 2011; Filippova & Altun, 2013, among others). Unfortu- nately, this makes such systems vulnerable to error propagation as there is no way to recover from an incorrect parse tree. With the state-of-the-art pars- ing systems achieving about 91 points in labeled attachment accuracy (Zhang & McDonald, 2014), the problem is not a negligible one. To our knowl- edge, there is no competitive compression system so far which does not require any linguistic pre- processing but tokenization. In this paper we research the following ques- tion: can a robust compression model be built which only uses tokens and has no access to syn- tactic or other linguistic information? While phe- nomena like long-distance relations may seem to make generation of grammatically correct com- pressions impossible, we are going to present an evidence to the contrary. In particular, we will present a model which benefits from the very re- cent advances in deep learning and uses word em- beddings and Long Short Term Memory models (LSTMs) to output surprisingly readable and in- formative compressions. Trained on a corpus of less than two million automatically extracted par- allel sentences and …

Content truncated to 500 words