Regularized Context Gates on Transformer for Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8555–8562 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8555 Regularized Context Gates on Transformer for Machine Translation Xintong Li1, Lemao Liu2, Rui Wang3, Guoping Huang2, Max Meng1 1The Chinese University of Hong Kong 2Tencent AI Lab 3National Institute of Information and Communications Technology znculee@gmail.com redmondliu@tencent.com wangrui@nict.go.jp donkeyhuang@tencent.com max.meng@cuhk.edu.hk Abstract Context gates are effective to control the con- tributions from the source and target contexts in the recurrent neural network (RNN) based neural machine translation (NMT). However, it is challenging to extend them into the advanced Transformer architecture, which is more complicated than RNN. This paper first provides a method to identify source and tar- get contexts and then introduce a gate mecha- nism to control the source and target contribu- tions in Transformer. In addition, to further re- duce the bias problem in the gate mechanism, this paper proposes a regularization method to guide the learning of the gates with supervi- sion automatically generated using pointwise mutual information. Extensive experiments on 4 translation datasets demonstrate that the pro- posed model obtains an averaged gain of 1.0 BLEU score over a strong Transformer base- line. 1 Introduction An essence to modeling translation is how to learn an effective context from a sentence pair. Statisti- cal machine translation (SMT) models the source context from the source-side of a translation model and models the target context from a target-side language model (Koehn et al., 2003; Koehn, 2009; Chiang, 2005). These two models are trained inde- pendently. On the contrary, neural machine transla- tion (NMT) advocates a unified manner to jointly learn source and target context using an encoder- decoder framework with an attention mechanism, leading to substantial gains over SMT in transla- tion quality (Sutskever et al., 2014; Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017). Prior work on attention mechanism (Luong et al., 2015; Liu et al., 2016; Mi et al., 2016; Chen et al., 2018; Li et al., 2018; Elbayad et al., 2018; Yang et al., 2020) have shown a better context represen- tation is helpful to translation performance. wǒ jīng cháng hé wǒ dè tóng háng mén yì qǐ tī qíu 。 我 经常 和 我的 同行 们 一起 踢 球 。 h Attention si Ioftenplaygolf with my colleagues . ti + zi + 1 − zi I often play golf with my colleagues . I often play soccer with my colleagues . Transformer: Context Gates: Regularized Context Gates: Figure 1: A running example to raise the context con- trol problem. Both original and context gated Trans- former obtain an unfaithful translation by wrongly translate “t¯i q´ıu” into “play golf” because referring too much target context. By regularizing the context gates, the purposed method corrects the translation of “t¯i q´ıu” into “play soccer”. The light font denotes the target words to be translated in the future. For original Trans- former, the source and target context are added directly without any …

Content truncated to 500 words