Variational Neural Machine Translation with Normalizing Flows

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7771–7777 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7771 Variational Neural Machine Translation with Normalizing Flows Hendra Setiawan Matthias Sperber Udhay Nallasamy Matthias Paulik Apple { hendra,sperber,udhay,mpaulik }@apple.com Abstract Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables. The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy. Unfortunately, learning informative latent variables is non-trivial, as the latent space can be prohibitively large, and the latent codes are prone to be ignored by many translation models at training time. Previous works impose strong assumptions on the distribution of the latent code and limit the choice of the NMT architecture. In this paper, we propose to apply the VNMT framework to the state-of-the-art Transformer and introduce a more flexible approximate posterior based on normalizing flows. We demonstrate the efficacy of our proposal under both in-domain and out-of-domain conditions, significantly outperforming strong baselines. 1 Introduction Translation is inherently ambiguous. For a given source sentence, there can be multiple plausible translations due to the author’s stylistic preference, domain, and other factors. On the one hand, the in- troduction of neural machine translation (NMT) has significantly advanced the field (Bahdanau et al., 2015), continually producing state-of-the-art translation accuracy. On the other hand, the exist- ing framework provides no explicit mechanisms to account for translation ambiguity. Recently, there has been a growing interest in latent-variable NMT (LV-NMT) that seeks to in- corporate latent random variables into NMT to ac- count for the ambiguities mentioned above. For instance, Zhang et al. (2016) incorporated latent codes to capture underlying global semantics of source sentences into NMT, while Su et al. (2018) proposed fine-grained latent codes at the word level. The learned codes, while not straightforward to an- alyze linguistically, are shown empirically to im- prove accuracy. Nevertheless, the introduction of latent random variables complicates the parameter estimation of these models, as it now involves in- tractable inference. In practice, prior work resorted to imposing strong assumptions on the latent code distribution, potentially compromising accuracy. In this paper, we focus on improving Variational NMT (VNMT) (Zhang et al., 2016): a family of LV-NMT models that relies on the amortized vari- ational method (Kingma and Welling, 2014) for inference. Our contributions are twofold. (1) We employ variational distributions based on normaliz- ing flows (Rezende and Mohamed, 2015), instead of uni-modal Gaussian. Normalizing flows can yield complex distributions that may better match the latent code’s true posterior. (2) We employ the Transformer architecture (Vaswani et al., 2017), including Transformer-Big, as our VNMT’s gen- erator network. We observed that the generator networks of most VNMT models belong to the RNN family that are relatively less powerful as a translation model than the Transformer. We demonstrate the efficacy of our proposal on the German-English IWSLT’14 and English- German WMT’18 tasks, giving …

Content truncated to 500 words