Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3082–3092, Osaka, Japan, December 11-17 2016. Improving Attention Modeling with Implicit Distortion and Fertility for Machine Translation Shi Feng 1 † Shujie Liu 2 Nan Yang 2 Mu Li 2 Ming Zhou 2 Kenny Q. Zhu 3 1 University of Maryland, College Park 2 Microsoft Research, Beijing, China 3 Shanghai Jiao Tong University, Shanghai, China shifeng@cs.umd.edu shujliu,nanya,muli,mingzhou@microsoft.com kzhu@cs.sjtu.edu.cn Abstract In neural machine translation, the attention mechanism facilitates the translation process by pro- ducing a soft alignment between the source sentence and the target sentence. However, without dedicated distortion and fertility models seen in traditional SMT systems, the learned alignment may not be accurate, which can lead to low translation quality. In this paper, we propose two novel models to improve attention-based neural machine translation. We propose a recurrent attention mechanism as an implicit distortion model, and a fertility conditioned decoder as an implicit fertility model. We conduct experiments on large-scale Chinese–English translation tasks. The results show that our models significantly improve both the alignment and translation quality compared to the original attention mechanism and several other variations. 1 Introduction Sequence-to-sequence neural machine translation (NMT) has shown promising results lately (Sutskever et al., 2014; Cho et al., 2014b). An NMT model typically consists of an encoding neural network which transforms the source sentence into some vector representation, and a decoding neural network which generates the target sentence from the vector representation. This is called the encoder-decoder model. In order to handle variable length inputs, recurrent neural networks (RNN) are usually used as the encoder and the decoder. The encoder RNN will read the words in the source sentence one by one and generate a sequence of corresponding hidden states; the decoder will then by conditioned on the encoder states to output each word in the target sentence. In (Cho et al., 2014b), only the last encoder state is used for target sentence generation, so the single hidden state vector must preserve all the necessary information in the source sentence for the decoding process , which is very difficult when the source sentence is long. To leverage the whole sequence of encoder states and retrieve information from the source sentence in a more flexible way, the attention mechanism (Bahdanau et al., 2014) was introduced into the encoder- decoder model. In an attention-based encoder-decoder model, matching scores between the source and target words are calculated based on their corresponding encoder and decoder states. These scores are then normalized and used as weights for the source words given each target word. This can be seen as a soft alignment and the attention mechanism here plays similar role to that of a traditional alignment model. In alignment models used in traditional machine translation models such as IBM Models (Brown et al., 1993), distortion and fertility are modeled explicitly. By comparison, in the attention mechanism, alignment is computed by matching the previous decoder hidden state with all the encoder hidden states, without …

Content truncated to 500 words