On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3449–3464 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3449 On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond Chen Wu1, Prince Zizhuang Wang2, William Yang Wang2 1Department of Foreign Languages and Literatures, Tsinghua University 2Department of Computer Science, University of California, Santa Barbara wu-c16@mails.tsinghua.edu.cn, zizhuang wang@ucsb.edu, william@cs.ucsb.edu Abstract Variational autoencoders (VAEs) combine la- tent variables with amortized variational infer- ence, whose optimization usually converges into a trivial local optimum termed posterior collapse, especially in text modeling. By track- ing the optimization dynamics, we observe the encoder-decoder incompatibility that leads to poor parameterizations of the data mani- fold. We argue that the trivial local optimum may be avoided by improving the encoder and decoder parameterizations since the posterior network is part of a transition map between them. To this end, we propose Coupled-VAE, which couples a VAE model with a determin- istic autoencoder with the same structure and improves the encoder and decoder parameter- izations via encoder weight sharing and de- coder signal matching. We apply the proposed Coupled-VAE approach to various VAE mod- els with different regularization, posterior fam- ily, decoder structure, and optimization strat- egy. Experiments on benchmark datasets (i.e., PTB, Yelp, and Yahoo) show consistently im- proved results in terms of probability estima- tion and richness of the latent space. We also generalize our method to conditional language modeling and propose Coupled-CVAE, which largely improves the diversity of dialogue gen- eration on the Switchboard dataset.1 1 Introduction The variational autoencoder (VAE) (Kingma and Welling, 2014) is a generative model that combines neural latent variables and amortized variational inference, which is efficient in estimating and sam- pling from the data distribution. It infers a posterior distribution for each instance with a shared infer- ence network and optimizes the evidence lower bound (ELBO) instead of the intractable marginal 1Our code is publicly available at https://github. com/ChenWu98/Coupled-VAE. log-likelihood. Given its potential to learn represen- tations from massive text data, there has been much interest in using VAE for text modeling (Zhao et al., 2017; Xu and Durrett, 2018; He et al., 2019). Prior work has observed that the optimization of VAE suffers from the posterior collapse prob- lem, i.e., the posterior becomes nearly identical to the prior and the decoder degenerate into a stan- dard language model (Bowman et al., 2016; Zhao et al., 2017). A widely mentioned explanation is that a strong decoder makes the collapsed poste- rior a good local optimum of ELBO, and existing solutions include weakened decoders (Yang et al., 2017; Semeniuta et al., 2017), modified regulariza- tion terms (Higgins et al., 2017; Wang and Wang, 2019), alternative posterior families (Rezende and Mohamed, 2015; Davidson et al., 2018), richer prior distributions (Tomczak and Welling, 2018), improved optimization strategies (He et al., 2019), and narrowed amortization gaps (Kim et al., 2018). In this paper, we provide a novel perspective for the posterior collapse problem. By comparing the optimization dynamics of VAE …

Content truncated to 500 words