Piecewise Latent Variables for Neural Variational Text Processing

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 422–432 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Piecewise Latent Variables for Neural Variational Text Processing Iulian V. Serban1∗and Alexander G. Ororbia II2∗and Joelle Pineau3 and Aaron Courville1 1 Department of Computer Science and Operations Research, Universite de Montreal 2College of Information Sciences & Technology, Penn State University 3School of Computer Science, McGill University iulian [DOT] vlad [DOT] serban [AT] umontreal [DOT] ca ago109 [AT] psu [DOT] edu jpineau [AT] cs [DOT] mcgill [DOT] ca aaron [DOT] courville [AT] umontreal [DOT] ca Abstract Advances in neural variational inference have facilitated the learning of power- ful directed graphical models with con- tinuous latent variables, such as varia- tional autoencoders. The hope is that such models will learn to represent rich, multi-modal latent factors in real-world data, such as natural language text. How- ever, current models often assume simplis- tic priors on the latent variables — such as the uni-modal Gaussian distribution — which are incapable of representing com- plex latent factors efficiently. To over- come this restriction, we propose the sim- ple, but highly flexible, piecewise constant distribution. This distribution has the ca- pacity to represent an exponential num- ber of modes of a latent target distribution, while remaining mathematically tractable. Our results demonstrate that incorporating this new latent distribution into different models yields substantial improvements in natural language processing tasks such as document modeling and natural language generation for dialogue. 1 Introduction The development of the variational autoencoder framework (Kingma and Welling, 2014; Rezende et al., 2014) has paved the way for learning large- scale, directed latent variable models. This has led to significant progress in a diverse set of machine learning applications, ranging from computer vi- sion (Gregor et al., 2015; Larsen et al., 2016) to natural language processing tasks (Mnih and Gre- gor, 2014; Miao et al., 2016; Bowman et al., 2015; ∗The first two authors contributed equally. Serban et al., 2017b). It is hoped that this frame- work will enable the learning of generative pro- cesses of real-world data — including text, audio and images — by disentangling and representing the underlying latent factors in the data. How- ever, latent factors in real-world data are often highly complex. For example, topics in newswire text and responses in conversational dialogue of- ten posses latent factors that follow non-linear (non-smooth), multi-modal distributions (i.e. dis- tributions with multiple local maxima). Nevertheless, the majority of current models as- sume a simple prior in the form of a multivariate Gaussian distribution in order to maintain mathe- matical and computational tractability. This is of- ten a highly restrictive and unrealistic assumption to impose on the structure of the latent variables. First, it imposes a strong uni-modal structure on the latent variable space; latent variable samples from the generating model (prior distribution) all cluster around a single mean. Second, it forces the latent variables to follow a perfectly symmet- ric distribution with constant kurtosis; this makes it difficult …

Content truncated to 500 words