Effective Approaches to Attention-based Neural Machine Translation

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Effective Approaches to Attention-based Neural Machine Translation Minh-Thang Luong Hieu Pham Christopher D. Manning Computer Science Department, Stanford University, Stanford, CA 94305 {lmthang,hyhieu,manning}@stanford.edu Abstract An attentional mechanism has lately been used to improve neural machine transla- tion (NMT) by selectively focusing on parts of the source sentence during trans- lation. However, there has been little work exploring useful architectures for attention-based NMT. This paper exam- ines two simple and effective classes of at- tentional mechanism: a global approach which always attends to all source words and a local one that only looks at a subset of source words at a time. We demonstrate the effectiveness of both approaches on the WMT translation tasks between English and German in both directions. With local attention, we achieve a significant gain of 5.0 BLEU points over non-attentional sys- tems that already incorporate known tech- niques such as dropout. Our ensemble model using different attention architec- tures yields a new state-of-the-art result in the WMT’15 English to German transla- tion task with 25.9 BLEU points, an im- provement of 1.0 BLEU points over the existing best system backed by NMT and an n-gram reranker.1 1 Introduction Neural Machine Translation (NMT) achieved state-of-the-art performances in large-scale trans- lation tasks such as from English to French (Luong et al., 2015) and English to German (Jean et al., 2015). NMT is appealing since it requires minimal domain knowledge and is conceptually simple. The model by Luong et al. (2015) reads through all the source words until the end-of-sentence symbol <eos> is reached. It then starts emitting one tar- get word at a time, as illustrated in Figure 1. NMT 1All our code and models are publicly available at http: //nlp.stanford.edu/projects/nmt. B C D <eos> X Y Z X Y Z <eos> A Figure 1: Neural machine translation – a stack- ing recurrent architecture for translating a source sequence A B C D into a target sequence X Y Z. Here, <eos> marks the end of a sentence. is often a large neural network that is trained in an end-to-end fashion and has the ability to general- ize well to very long word sequences. This means the model does not have to explicitly store gigantic phrase tables and language models as in the case of standard MT; hence, NMT has a small memory footprint. Lastly, implementing NMT decoders is easy unlike the highly intricate decoders in stan- dard MT (Koehn et al., 2003). In parallel, the concept of “attention” has gained popularity recently in training neural networks, al- lowing models to learn alignments between dif- ferent modalities, e.g., between image objects and agent actions in the dynamic control problem (Mnih et al., 2014), between speech frames and text in the speech recognition task (Chorowski et al., 2014), or between visual features of a picture and its text description in the image caption gen- eration …

Content truncated to 500 words