Improving Neural Machine Translation by Achieving Knowledge Transfer with Sentence Alignment Learning

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 260–270 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 260 Improving Neural Machine Translation by Achieving Knowledge Transfer with Sentence Alignment Learning Xuewen Shi1,2, Heyan Huang1,2, Wenguan Wang1,3, Ping Jian1,2∗, Yi-Kun Tang1,2 1School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China 2Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications 3Inception Institute of Artificial Intelligence, UAE {xwshi, hhy63, wenguanwang, pjian, tangyk}@bit.edu.cn Abstract Neural Machine Translation (NMT) optimized by Maximum Likelihood Estimation (MLE) lacks the guarantee of translation adequacy. To alleviate this problem, we propose an NMT approach that heightens the adequacy in ma- chine translation by transferring the seman- tic knowledge learned from bilingual sentence alignment. Specifically, we first design a discriminator that learns to estimate sentence aligning score over translation candidates, and then the learned semantic knowledge is trans- fered to the NMT model under an adversarial learning framework. We also propose a gated self-attention based encoder for sentence em- bedding. Furthermore, an N-pair training loss is introduced in our framework to aid the dis- criminator in better capturing lexical evidence in translation candidates. Experimental results show that our proposed method outperforms baseline NMT models on Chinese-to-English and English-to-German translation tasks. Fur- ther analysis also indicates the detailed seman- tic knowledge transfered from the discrimina- tor to the NMT model. 1 Introduction Recently, with the renaissance of deep learn- ing, end-to-end Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014a; Sutskever et al., 2014; Bahdanau et al., 2014) has gained remarkable perfor- mance (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017). Early NMT solutions are typically optimized to maximize the likelihood es- timation (MLE) of each word in the ground truth translations during the training procedure. How- ever, such an objective cannot guarantee the suffi- ciency of the generated translations in the NMT model, due to the lack of quantitative measure- ∗Corresponding author: Ping Jian yi zhi mao xiao pa zai cao shang a little lays cat on grass pitch a small lays cat on a straw mat Source NMT1 NMT2 dian Figure 1: Comparison between two Chinese-to- English translation examples of two independent NMT systems. Lines between Source and NMTs rep- resent model generated alignments (each source word cannot be covered more than once). Words in boxes are key words and red dotted dashed boxes indicate in- correct translations. Based on the model generated at- tention weights, NMT2 covers more source words than NMT1, which is opposite to human judgments. ment for the information transformational com- pleteness from the source side to the target side. Some existing work alleviates this problem by directly incorporating coverage or fertility mech- anisms to an NMT model (Tu et al., 2016; Feng et al., 2016; Kong et al., 2019). However, the prob- lem is that attention weights based coverage calcu- lation for NMT is insensitive and sometimes even inaccurate to translation errors. Furthermore, …

Content truncated to 500 words