Unsupervised Sentence Compression using Denoising Auto-Encoders

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 413–422 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 413 Unsupervised Sentence Compression using Denoising Auto-Encoders Thibault Fevry∗ Center for Data Science New York University Thibault.Fevry@nyu.edu Jason Phang∗ Center for Data Science New York University jasonphang@nyu.edu Abstract In sentence compression, the task of short- ening sentences while retaining the original meaning, models tend to be trained on large corpora containing pairs of verbose and compressed sentences. To remove the need for paired corpora, we emulate a summarization task and add noise to extend sentences and train a denoising auto-encoder to recover the original, constructing an end-to-end training regime without the need for any examples of compressed sentences. We conduct a human evaluation of our model on a standard text summarization dataset and show that it performs comparably to a supervised base- line based on grammatical correctness and retention of meaning. Despite being exposed to no target data, our unsupervised models learn to generate imperfect but reasonably readable sentence summaries. Although we underperform supervised models based on ROUGE scores, our models are competitive with a supervised baseline based on human evaluation for grammatical correctness and retention of meaning. 1 Introduction Sentence compression is the task of condensing a longer sentence into a shorter one that still re- tains the meaning of the original. Past models for sentence compression have tended to rely heavily on strong linguistic priors such as syntactic rules or heuristics (Dorr et al., 2003; Cohn and Lap- ata, 2008). More recent work using deep learning involves models trained without strong linguistic priors, instead requiring large corpora consisting of pairs of longer and shorter sentences (Miao and Blunsom, 2016). ∗ Denotes equal contribution Sentence compression can also be can be seen as a “scaled down version of the text summariza- tion problem” (Knight and Marcu, 2002). Within text summarization, two broad approaches exist: extractive approaches extract explicit tokens or phrases from the reference text, whereas abstrac- tive approaches involve a compressed paraphras- ing of the reference text, similar to the approach humans might take (Jing, 2000, 2002). In the related domain of machine translation, a task that also involves learning a mapping from one string of tokens to another, state of the art models using deep learning techniques are trained on large parallel corpora. Recent promising work on unsupervised neural machine translation (Artetxe et al., 2017; Lample et al., 2017) has shown that with the right training regime, it is possible to train models for machine translation between two languages given only two unpaired monolingual corpora. In this paper, we apply neural text summariza- tion techniques to the task of sentence compres- sion, focusing on on extractive summarization. However, we depart significantly from prior work by taking a fully unsupervised training approach. Beyond not using parallel corpora, we train our model using a single corpus. In contrast to un- supervised neural machine translation, which still uses two corpora, we do not …

Content truncated to 500 words