Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 319–328, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Language as a Latent Variable: Discrete Generative Models for Sentence Compression Yishu Miao1, Phil Blunsom1,2 1University of Oxford, 2Google Deepmind {yishu.miao, phil.blunsom}@cs.ox.ac.uk Abstract In this work we explore deep generative mod- els of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence con- ditioned on this latent summary. In our em- pirical evaluation we show that generative for- mulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compres- sion scenarios where we show that it is possi- ble to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data. 1 Introduction The recurrent sequence-to-sequence paradigm for natural language generation (Kalchbrenner and Blun- som, 2013; Sutskever et al., 2014) has achieved re- markable recent success and is now the approach of choice for applications such as machine transla- tion (Bahdanau et al., 2015), caption generation (Xu et al., 2015) and speech recognition (Chorowski et al., 2015). While these models have developed so- phisticated conditioning mechanisms, e.g. attention, fundamentally they are discriminative models trained only to approximate the conditional output distribu- tion of strings. In this paper we explore modelling the joint distribution of string pairs using a deep genera- tive model and employing a discrete variational auto- encoder (VAE) for inference (Kingma and Welling, 2014; Rezende et al., 2014; Mnih and Gregor, 2014). We evaluate our generative approach on the task of sentence compression. This approach provides both alternative supervised objective functions and the opportunity to perform semi-supervised learning by exploiting the VAEs ability to marginalise the latent compressed text for unlabelled data. Auto-encoders (Rumelhart et al., 1985) are a typi- cal neural network architecture for learning compact data representations, with the general aim of perform- ing dimensionality reduction on embeddings (Hinton and Salakhutdinov, 2006). In this paper, rather than seeking to embed inputs as points in a vector space, we describe them with explicit natural language sen- tences. This approach is a natural fit for summarisa- tion tasks such as sentence compression. According to this, we propose a generative auto-encoding sen- tence compression (ASC) model, where we intro- duce a latent language model to provide the variable- length compact summary. The objective is to perform Bayesian inference for the posterior distribution of summaries conditioned on the observed utterances. Hence, in the framework of VAE, we construct an in- ference network as the variational approximation of the posterior, which generates compression samples to optimise the variational lower bound. The most common family of variational …

Content truncated to 500 words