Supervised Attentions for Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2283–2288, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Supervised Attentions for Neural Machine Translation Haitao Mi Zhiguo Wang Abe Ittycheriah T.J. Watson Research Center IBM 1101 Kitchawan Rd, Yorktown Heights, NY 10598 {hmi, zhigwang, abei}@us.ibm.com Abstract In this paper, we improve the attention or alignment accuracy of neural machine trans- lation by utilizing the alignments of train- ing sentence pairs. We simply compute the distance between the machine attentions and the “true” alignments, and minimize this cost in the training procedure. Our experi- ments on large-scale Chinese-to-English task show that our model improves both transla- tion and alignment qualities significantly over the large-vocabulary neural machine transla- tion system, and even beats a state-of-the-art traditional syntax-based system. 1 Introduction Neural machine translation (NMT) has gained pop- ularity in recent two years (e.g. (Bahdanau et al., 2014; Jean et al., 2015; Luong et al., 2015; Mi et al., 2016b; Li et al., 2016), especially for the attention- based models of Bahdanau et al. (2014). The attention model plays a crucial role in NMT, as it shows which source word(s) the model should focus on in order to predict the next target word. However, the attention or alignment quality of NMT is still very low (Mi et al., 2016a; Tu et al., 2016). In this paper, we alleviate the above issue by uti- lizing the alignments (human annotated data or ma- chine alignments) of the training set. Given the alignments of all the training sentence pairs, we add an alignment distance cost to the objective func- tion. Thus, we not only maximize the log translation probabilities, but also minimize the alignment dis- tance cost. Large-scale experiments over Chinese- to-English on various test sets show that our best method for a single system improves the transla- tion quality significantly over the large vocabulary NMT system (Section 5) and beats the state-of-the- art syntax-based system. 2 Neural Machine Translation As shown in Figure 1, attention-based NMT (Bah- danau et al., 2014) is an encoder-decoder network. the encoder employs a bi-directional recurrent neu- ral network to encode the source sentence x = (x1, ..., xl), where l is the sentence length (includ- ing the end-of-sentence ⟨eos⟩), into a sequence of hidden states h = (h1, ..., hl), each hi is a concate- nation of a left-to-right −→ hi and a right-to-left ←− hi. Given h, the decoder predicts the target transla- tion by maximizing the conditional log-probability of the correct translation y∗ = (y∗ 1, ...y∗ m), where m is the sentence length (including the end-of- sentence). At each time t, the probability of each word yt from a target vocabulary Vy is: p(yt|h, y∗ t−1..y∗ 1) = g(st, y∗ t−1), (1) where g is a two layer feed-forward neural network over the embedding of the previous word y∗ t−1, and the hidden state st. The st is computed as: st = q(st−1, y∗ t−1, Ht) (2) Ht = ��l …

Content truncated to 500 words