Neural Machine Translation with Decoding History Enhanced Attention

Proceedings of the 27th International Conference on Computational Linguistics, pages 1464–1473 Santa Fe, New Mexico, USA, August 20-26, 2018. 1464 Neural Machine Translation with Decoding-History Enhanced Attention Mingxuan Wang1 Jun Xie1 Zhixing Tan1 Jinsong Su2 Deyi Xiong3 Chao bian1 1Mobile Internet Group, Tencent Technology Co., Ltd {xuanswang,stiffxie,zhixingtan,chaobian}@tencent.com 2Xiamen University, Xiamen, China {jssu}@xmu.edu.cn 3Soochow University, Suzhou, China {dyxiong}@suda.edu.cn Abstract Neural Machine Translation (NMT) with source side attention have achieved remarkable perfor- mance. however, there has been little work exploring to attend to the target side which can po- tentially enhance the memory capbility of NMT. We reformulate a Decoding-History Enhanced Attention mechanism (DHEA) to render NMT model better at selecting both source side and tar- get side information. DHEA enables a dynamic control on the ratios at which source and target contexts contribute to the generation of target words, offering a way to weakly induce struc- ture relations among both source and target tokens. It also allows training errors to be directly back-propagated through short-cut connections and effectively alleviates the gradient vanishing problem. The empirical study on Chinese-English translation shows that our model with proper configuration can improve by 0.9 BLEU upon Transformer and achieve the best reported results in the same dataset. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art NMT systems. 1 Introduction Neural Machine Translation(Sutskever et al., 2014) generally adopts an encoder-decoder framework, where the encoder reads and encodes the input text into a distributed representation and the decoder generates translated text conditioned on the input representation. A potential issue with this encoder- decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector. This may make it difficult for the neural network to cope with translating long sentences. A recent successful extension of NMT models is the attention mechanism which conducts a soft search over source tokens and yields an attentive vector to represent the most relevant segments of the source sentence for the current decoding state (Bahdanau et al., 2014; Jean et al., 2015; Luong et al., 2014; Vaswani et al., 2017). The typical attention mechanism frees the neural translation model from having to squash all the infor- mation of the source sentence into a fixed vector, however, it ignores some important information hidden in the target sequence (Cheng et al., 2016). At least three challenges still remains in the translation pro- cess. The first issue relates to the memory compression problems in the decoding process. During each translation step, a hidden state vector implicitly maintains at least two types of information, including both the most relevant source contexts and the language model over the partially translated sentence. The memory capacity of a single dense vector is not powerful enough to store these information. The sec- ond challenge comes with the training and optimization problem associated with vanishing or exploding gradients, which has been studied in the context of vanilla RNNs. Although …

Content truncated to 500 words