Almost Free Semantic Draft for Neural Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3931–3941 June 6–11, 2021. ©2021 Association for Computational Linguistics 3931 Almost Free Semantic Draft for Neural Machine Translation Xi Ai College of Computer Science Chongqing University barid.x.ai@gmail.com Bin Fang College of Computer Science Chongqing University fb@cqu.edu.cn Abstract Translation quality can be improved by global information from the required target sentence because the decoder can understand both past and future information. However, the model needs additional cost to produce and consider such global information. In this work, to in- ject global information but also save cost, we present an efficient method to sample and con- sider a semantic draft as global information from semantic space for decoding with almost free of cost. Unlike other successful adapta- tions, we do not have to perform an EM-like process that repeatedly samples a possible se- mantic from the semantic space. Empirical ex- periments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in infer- ence efficiency. We will open all our source code on GitHub. 1 Introduction Successful NMT (Neural Machine Translation) (Vaswani et al., 2017; Bahdanau et al., 2015; John- son et al., 2017; Ng et al., 2019) can translate sen- tences through left to right or through right to left. However, there is one critical limitation in this di- agram. That is, the decoder can only have access to directional information (left-to-right or right- to-left) when processing auto-regressive (Graves, 2013). To alleviate this pain, there have been three suc- cessful lines. 1) Generative NMT: (Zheng et al., 2020; Shah and Barber, 2018; Su et al., 2018; Zhang et al., 2016; Eikema and Aziz, 2019) adapt VAE (variational auto-encoder) (Altieri and Duve- naud, 2015; Kingma and Ba, 2015; Bowman et al., 2016) for NMT that is trained in generative model settings, modeling the semantics of the source and target sentences in latent space. 2) Deliberation: since the problem is caused by the one-pass process of decoding in the auto-regression process, (Xia et al., 2017) present a framework to predict a guess target sentence in the first-pass and jointly consid- ers the encoding and the guess target sentence in the second-pass. 3) Soft-prototype: (Wang et al., 2019) present a framework to generate a prototype on the encoder side and then the decoder can jointly use the encoding and the prototype. Although em- pirical results show the previous methods can suc- cessfully inject global information into the decoder, these methods either introduce computational com- plexity to the encoder-decoder architecture or em- ploy an EM-like process in inferring, thus requiring even more than 100% additional time to produce and consider global information in inferring. In this work, we present an efficient method to sample and consider a semantic draft as global in- formation for decoding with almost free of cost, following the line of generative NMT. Concretely, we sample the semantic draft from semantic space that is a …

Content truncated to 500 words