Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8512–8525 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8512 Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine Translation Arya D. McCarthy♠♦ and Xian Li♦ and Jiatao Gu♦ and Ning Dong♦ ♠Johns Hopkins University ♦Facebook arya@jhu.edu, {xianl,jgu,dnn}@fb.com Abstract This paper proposes a simple and effective approach to address the problem of poste- rior collapse in conditional variational autoen- coders (CVAEs). It thus improves perfor- mance of machine translation models that use noisy or monolingual data, as well as in con- ventional settings. Extending Transformer and conditional VAEs, our proposed latent vari- able model measurably prevents posterior col- lapse by (1) using a modified evidence lower bound (ELBO) objective which promotes mu- tual information between the latent variable and the target, and (2) guiding the latent vari- able with an auxiliary bag-of-words predic- tion task. As a result, the proposed model yields improved translation quality compared to existing variational NMT models on WMT Ro↔En and De↔En. With latent variables being effectively utilized, our model demon- strates improved robustness over non-latent Transformer in handling uncertainty: exploit- ing noisy source-side monolingual data (up to +3.2 BLEU), and training with weakly aligned web-mined parallel data (up to +4.7 BLEU). 1 Introduction The conditional variational autoencoder (CVAE; Sohn et al., 2015) is a conditional generative model for structured prediction tasks like machine translation. This model, learned by variational Bayesian methods (Kingma and Welling, 2014), can capture global signal about the target in its latent variables. Unfortunately, variational inference for text generation often yields models that ignore their latent variables (Bowman et al., 2016), a phenomenon called posterior collapse. In this paper, we introduce a new loss func- tion for CVAEs that counteracts posterior collapse, motivated by our analysis of CVAE’s evidence lower bound objective (ELBO). Our analysis (§2) reveals that optimizing ELBO’s second term not only brings the variational posterior approximation closer to the prior, but also decreases mutual infor- mation between latent variables and observed data. Based on this insight, we modify CVAE’s ELBO in two ways (§3): (1) We explicitly add a principled mutual information term back into the training ob- jective, and (2) we use a factorized decoder (Chen et al., 2017), which also predicts the target bag- of-words as an auxiliary decoding distribution to regularize our latent variables. Our objective is effective even without Kullback–Leibler term (KL) annealing (Bowman et al., 2016), a strategy for it- eratively altering ELBO over the course of training to avoid posterior collapse. In applying our method to neural machine trans- lation (NMT; Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014), we find that we have mea- surably mitigated posterior collapse. The latent variables are not ignored, even in the presence of a powerful Transformer decoder. By addressing this problem, the resulting NMT model has improved robustness and performance in low-resource sce- narios. Noisy data like those scraped from the Internet (Smith et al., 2013; Michel and Neubig, …

Content truncated to 500 words