A Batch Normalized Inference Network Keeps the KL Vanishing Away

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2636–2649 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2636 A Batch Normalized Inference Network Keeps the KL Vanishing Away Qile Zhu1, Wei Bi2, Xiaojiang Liu2, Xiyao Ma1, Xiaolin Li3 and Dapeng Wu1 1University of Florida, 2Tencent AI Lab, 3AI Institute, Tongdun Technology {valder,maxiy,dpwu}@ufl.edu {victoriabi,kieranliu}@tencent.com xiaolin.li@tongdun.net Abstract Variational Autoencoder (VAE) is widely used as a generative model to approximate a model’s posterior on latent variables by com- bining the amortized variational inference and deep neural networks. However, when paired with strong autoregressive decoders, VAE of- ten converges to a degenerated local optimum known as “posterior collapse”. Previous ap- proaches consider the Kullback–Leibler diver- gence (KL) individual for each datapoint. We propose to let the KL follow a distribution across the whole dataset, and analyze that it is sufficient to prevent posterior collapse by keep- ing the expectation of the KL’s distribution positive. Then we propose Batch Normalized- VAE (BN-VAE), a simple but effective ap- proach to set a lower bound of the expectation by regularizing the distribution of the approxi- mate posterior’s parameters. Without introduc- ing any new model component or modifying the objective, our approach can avoid the pos- terior collapse effectively and efficiently. We further show that the proposed BN-VAE can be extended to conditional VAE (CVAE). Em- pirically, our approach surpasses strong autore- gressive baselines on language modeling, text classification and dialogue generation, and ri- vals more complex approaches while keeping almost the same training time as VAE. 1 Introduction Variational Autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014)is one of the most popular generative framework to model com- plex distributions. Different from the Autoencoder (AE), VAE provides a distribution-based latent rep- resentation for the data, which encodes the input x into a probability distribution z and reconstructs the original input using samples from z. When *This work was done when Qile Zhu was an intern at Tencent AI Lab. Wei Bi is the corresponding author. inference, VAE first samples the latent variable from the prior distribution and then feeds it into the decoder to generate an instance. VAE has been successfully applied in many NLP tasks, including topic modeling (Srivastava and Sutton, 2017; Miao et al., 2016; Zhu et al., 2018), language modeling (Bowman et al., 2016), text generation (Zhao et al., 2017b) and text classification (Xu et al., 2017). An autoregressive decoder (e.g., a recurrent neu- ral network) is a common choice to model the text data. However, when paired with strong au- toregressive decoders such as LSTMs (Hochreiter and Schmidhuber, 1997) and trained under conven- tional training strategy, VAE suffers from a well- known problem named the posterior collapse or the KL vanishing problem. The decoder in VAE learns to reconstruct the data independent of the latent variable z, and the KL vanishes to 0. Many convincing solutions have been proposed to prevent posterior collapse. Among them, fixing the KL as a positive constant is an important …

Content truncated to 500 words