Target Foresight Based Attention for Neural Machine Translation

Proceedings of NAACL-HLT 2018, pages 1380–1390 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Target Foresight based Attention for Neural Machine Translation∗ Xintong Li†, Lemao Liu‡, Zhaopeng Tu‡, Shuming Shi‡, Max Meng† †The Chinese University of Hong Kong {xtli, qhmeng}@ee.cuhk.edu.hk ‡Tencent AI Lab {redmondliu, zptu, shumingshi}@tencent.com Abstract in neural machine translation, an attention model is used to identify the aligned source words for a target word （target foresight word）in order to select translation con- text, but it does not make use of any in- formation of this target foresight word at all. previous work proposed an approach to improve the attention model by explic- itly accessing this target foresight word and demonstrated the substantial gains in alignment task. however, this approach is useless in machine translation task on which the target foresight word is unavail- able. in this paper, we propose a new attention model enhanced by the implicit information of target foresight word ori- ented to both alignment and translation tasks. empirical experiments on chinese- to-english and japanese-to-english datasets show that the proposed attention model delivers significant improvements in terms of both alignment error rate and bleu. 1 Introduction Since neural machine translation (NMT) was proposed (Bahdanau et al., 2014), it has been attracted increasing interests in machine translation community (Luong et al., 2015b; Tu et al., 2016; Feng et al., 2016; Cohn et al., 2016). NMT not only yields impressive translation performance in practice, but also has appealing model architecture in essence. Compared with traditional statistical machine translation (Koehn et al., 2003; Chiang, 2005), one of advantages in NMT is that its archi- tecture combines language model, translation model and alignment between source and tar- get words in a unified manner rather than a ∗Work done when X. Li interning at Tencent AI Lab. L. Liu is the corresponding author. fă guó shī yè rén shù zài dù huí shēng </S> 法国 失业 人数 再度 回升 </S> French unemployment rate rises again </S> (a) Baseline fă guó shī yè rén shù zài dù huí shēng </S> 法国 失业 人数 再度 回升 </S> French unemployment rises again </S> JJ NN VBZ RB EOS (b) TFA-NMT French unemployment rises again </S> (c) Reference Figure 1: A running example to motivate the pro- posed model. (a) The baseline obtains a transla- tion error due to the incorrect attention. (b) With the help of the target foresight information “VBZ”, TFA-NMT is likely to figure out the exact transla- tion as the reference in (c). The light font denotes the target words to be translated in future. Both dashed or solid arrowed lines denote the alignments and solid one denotes the 1-best alignment. pipeline manner, and it thereby has the poten- tial to alleviate the issue of error propagation. In NMT, the attention mechanism plays an important role. It calculates the alignments of a target word with respect to the source words for translation context selection. Although the source words are always available in inference, the target word, …

Content truncated to 500 words