Neural Machine Translation with Supervised Attention

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3093–3102, Osaka, Japan, December 11-17 2016. Neural Machine Translation with Supervised Attention Lemao Liu, Masao Utiyama, Andrew Finch and Eiichiro Sumita National Institute of Information and Communications Technology (NICT) 3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto, Japan {lmliu,first.last}@nict.go.jp Abstract The attention mechanism is appealing for neural machine translation, since it is able to dynam- ically encode a source sentence by generating a alignment between a target word and source words. Unfortunately, it has been proved to be worse than conventional alignment models in alignment accuracy. In this paper, we analyze and explain this issue from the point view of re- ordering, and propose a supervised attention which is learned with guidance from conventional alignment models. Experiments on two Chinese-to-English translation tasks show that the super- vised attention mechanism yields better alignments leading to substantial gains over the standard attention based NMT. 1 Introduction Neural Machine Translation (NMT) has achieved great successes on machine translation tasks recently (Bahdanau et al., 2015; Sutskever et al., 2015). Generally, it relies on a recurrent neural network under the Encode-Decode framework: it firstly encodes a source sentence into context vectors and then generates its translation token-by-token, selecting from the target vocabulary. Among different variants of NMT, attention based NMT, which is the focus of this paper,1 is attracting increasing interests in the community (Bahdanau et al., 2015; Luong et al., 2015). One of its advantages is that it is able to dynamically make use of the encoded context through an attention mechanism thereby allowing the use of fewer hidden layers while still maintaining high levels of translation performance. An attention mechanism is designed to predict the alignment of a target word with respect to source words (Bahdanau et al., 2015). In order to facilitate incremental decoding, it tries to make this alignment prediction without the information about the target word itself, and thus this attention can be considered to be a form of a reordering model (see §2 for more details). In contrast, conventional alignment models are able to use the target word to infer its alignments (Och and Ney, 2000; Dyer et al., 2013; Liu and Sun, 2015), and as a result there is a substantial gap in quality between the alignments derived by this attention based NMT and conventional alignment models (54 VS 30 in terms of AER for Chinese-to-English as reported in (Cheng et al., 2016)). This discrepancy might be an indication that the potential of attention- based NMT is limited. In addition, the attention in NMT is learned in an unsupervised manner without explicit prior knowledge about alignment.2 However, in conventional statistical machine translation (SMT), it is standard practice to learn reordering models in a supervised manner with the guidance from conventional alignment models (Xiong et al., 2006; Koehn et al., 2007; Bisazza and Federico, 2016). Inspired by the supervised reordering in conventional SMT, in this paper, we propose a Supervised Attention based NMT (SA-NMT) model. Specifically, similar to …

Content truncated to 500 words