Effective Estimation of Deep Generative Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7220–7236 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7220 Effective Estimation of Deep Generative Language Models Tom Pelsmaeker ILCC University of Edinburgh t.l.pelsmaeker@sms.ed.ac.uk Wilker Aziz ILLC University of Amsterdam w.aziz@uva.nl Abstract Advances in variational inference enable pa- rameterisation of probabilistic models by deep neural networks. This combines the statisti- cal transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language mod- elling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierar- chical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel tech- niques, and extensions to the model. To es- tablish a ranking of techniques, we perform a systematic comparison using Bayesian optimi- sation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on conve- nience. We also make several empirical obser- vations and recommendations of best practices that should help researchers interested in this exciting field. 1 Introduction Deep generative models (DGMs) are probabilis- tic latent variable models parameterised by neural networks (NNs). Specifically, DGMs optimised with amortised variational inference and reparam- eterised gradient estimates (Kingma and Welling, 2014; Rezende et al., 2014), better known as vari- ational auto-encoders (VAEs), have spurred much interest in various domains, including computer vision and natural language processing (NLP). In NLP, VAEs have been developed for word representation (Rios et al., 2018), morphological analysis (Zhou and Neubig, 2017), syntactic and Work done while the first author was at the University of Amsterdam. Code is available at https://github.com/ tom-pelsmaeker/deep-generative-lm semantic parsing (Corro and Titov, 2018; Lyu and Titov, 2018), document modelling (Miao et al., 2016), summarisation (Miao and Blunsom, 2016), machine translation (Zhang et al., 2016; Schulz et al., 2018; Eikema and Aziz, 2019), language and vision (Pu et al., 2016; Wang et al., 2017), dialogue modelling (Wen et al., 2017; Serban et al., 2017), speech modelling (Fraccaro et al., 2016), and, of course, language modelling (Bowman et al., 2016; Goyal et al., 2017). One problem remains common to the majority of these models, VAEs often learn to ignore the latent variables. We investigate this problem, dubbed posterior collapse, in the context of language models (LMs). In a deep generative LM (Bowman et al., 2016), sentences are generated conditioned on samples from a continuous latent space, an idea with vari- ous practical applications. For example, one can constrain this latent space to promote generalisa- tions that are in line with linguistic knowledge and intuition (Xu and Durrett, 2018). This also allows for greater flexibility in how the model is used, for example, to generate sentences that live—in la- tent space—in a neighbourhood of a given observa- tion (Bowman et al., 2016). Despite …

Content truncated to 500 words