Variational Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 521–530, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Variational Neural Machine Translation Biao Zhang1,2, Deyi Xiong1∗, Jinsong Su2, Hong Duan2 and Min Zhang1 Provincial Key Laboratory for Computer Information Processing Technology Soochow University, Suzhou, China 2150061 Xiamen University, Xiamen, China 3610052 zb@stu.xmu.edu.cn, {jssu,hduan}@xmu.edu.cn {dyxiong, minzhang}@suda.edu.cn Abstract Models of neural machine translation are of- ten from a discriminative family of encoder- decoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational model to learn this conditional distribution for neu- ral machine translation: a variational encoder- decoder model that can be trained end-to-end. Different from the vanilla encoder-decoder model that generates target translations from hidden representations of source sentences alone, the variational model introduces a con- tinuous latent variable to explicitly model un- derlying semantics of source sentences and to guide the generation of target translations. In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to esti- mate the variational lower bound. Experi- ments on both Chinese-English and English- German translation tasks show that the pro- posed variational neural machine translation achieves significant improvements over the vanilla neural machine translation baselines. 1 Introduction Neural machine translation (NMT) is an emerging translation paradigm that builds on a single and unified end-to-end neural network, instead of us- ing a variety of sub-models tuned in a long training pipeline. It requires a much smaller memory than ∗Corresponding author phrase- or syntax-based statistical machine transla- tion (SMT) that typically has a huge phrase/rule ta- ble. Due to these advantages over traditional SMT system, NMT has recently attracted growing inter- ests from both deep learning and machine transla- tion community (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015a; Luong et al., 2015b; Shen et al., 2015; Meng et al., 2015; Tu et al., 2016). Current NMT models mainly take a discrimi- native encoder-decoder framework, where a neu- ral encoder transforms source sentence x into dis- tributed representations, and a neural decoder gen- erates the corresponding target sentence y according to these representations1 (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014). Typically, the underlying semantic representations of source and target sentences are learned in an implicit way in this framework, which heavily relies on the atten- tion mechanism (Bahdanau et al., 2014) to iden- tify semantic alignments between source and target words. Due to potential errors in these alignments, the attention-based context vector may be insuffi- cient to capture the entire meaning of a source sen- tence, hence resulting in undesirable translation phe- nomena (Tu et al., 2016). Unlike the vanilla encoder-decoder framework, we model underlying semantics of bilingual sen- tence pairs explicitly. We assume that there exists a continuous latent variable …

Content truncated to 500 words